Kilobyte Models: Neural Networks as a Seed and a Quantized Latent

# Kilobyte Models: Neural Networks as a Seed and a Quantized Latent
Source: [https://arxiv.org/html/2608.00860](https://arxiv.org/html/2608.00860)
###### Abstract

The cost of storing and transmitting a trained neural network scales with its parameter count, a bottleneck for over\-the\-air updates, on\-device libraries, and other bandwidth\-bound deployments\. We study an extreme form of model compression in which the deployable artifact is not the weights but a short recipe for regenerating them\. Building on Mapping Networks, which express a network’s weights as a nonlinear function of a compact trainable latent and a fixed random basis, we observe that only the latent need be stored, because the basis and initialization center are reproducible from an integer seed\. A model becomes a seed together with a quantized latent, whose size is set by the latent dimension and bit width rather than the parameter count\. We formalize this artifact and introduce a seeded block\-wise basis that scales to networks whose projection cannot be held in memory\. In our experiments, a mapped model is as accurate as the same network quantized aggressively to a few bits per weight, while taking far fewer bytes to store\. Reaching the most aggressive bit widths depends on fine\-tuning the latent with quantization in the loop\. The results do not depend on the particular random basis, and a structured basis lets the weights be regenerated almost for free even for large networks\.

## 1Introduction

Storing and transmitting neural networks scales with parameter count, creating bottlenecks for bandwidth\-constrained deployments like edge devices, over\-the\-air updates, and massively scaled adapters\. In these scenarios, the binding constraint is storage size rather than inference cost\. Standard compression techniques, such as pruning\(Hanet al\.[2015](https://arxiv.org/html/2608.00860#bib.bib1); Frankle and Carbin[2019](https://arxiv.org/html/2608.00860#bib.bib2)\), quantization\(Jacobet al\.[2018](https://arxiv.org/html/2608.00860#bib.bib3); Hanet al\.[2016](https://arxiv.org/html/2608.00860#bib.bib4)\), low\-rank factorization, and distillation\(Hintonet al\.[2015](https://arxiv.org/html/2608.00860#bib.bib5)\), address this by shrinking the description of the original weight tensors\.

Alternatively, one can store a compact recipe to regenerate weights\. Neural networks have low intrinsic dimensionality\(Denilet al\.[2013](https://arxiv.org/html/2608.00860#bib.bib6)\), allowing optimization within random, low\-dimensional subspaces\(Liet al\.[2018](https://arxiv.org/html/2608.00860#bib.bib7)\)\. If a random seed generates this subspace, the model is entirely defined by that seed and a small coordinate vector, reducing the byte count\.

Building on Mapping Networks\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\), which represent weights as a nonlinear function of a small trainable latent vector and a fixed random basis, we propose an extreme compression paradigm\. Since the random basis and initialization are deterministically regenerated from an integer seed, only the latent vector requires storage\. Quantizing this latent to a few bits shrinks the entire artifact to a few kilobytes\. We formalize this deployable artifact as a*Kilobyte model*and evaluate its accuracy at extreme compression rates\.

Our specific contributions are as follows:

- •We formalize the Kilobyte model artifact \(seed, quantized latent, and normalization parameters\), which is sized by latent dimensionddand bit widthbbrather than the full network parameter countPP\(Section[4](https://arxiv.org/html/2608.00860#S4.SSx1)\)\.
- •We introduce a scalable, seeded, block\-wise random basis that avoids materializing dense projection matrices, regenerating weights efficiently across platforms \(Section[4](https://arxiv.org/html/2608.00860#S4.SSx2)\)\.
- •We extend this parameterization to*fine\-tuning*, using the latent as a compressed task adapter over a shared pretrained base network to produce a small\-sized delta \(Section[4](https://arxiv.org/html/2608.00860#S4.SSx4)\)\.
- •We demonstrate that Kilobyte models match aggressively quantized networks at a fraction of their size\. For instance, a 4\-bit model achieves98\.6%98\.6\\%on MNIST in just 2 KB, yielding a26×26\\timesreduction over the quantized baseline \(Section[5](https://arxiv.org/html/2608.00860#S5.SSx2)\)\.

## 2Related Work

#### Random subspaces and intrinsic dimension\.

Neural networks can be successfully optimized within random, low\-dimensional affine subspaces, revealing a surprisingly small intrinsic dimension for most objective landscapes\(Liet al\.[2018](https://arxiv.org/html/2608.00860#bib.bib7); Aghajanyanet al\.[2021](https://arxiv.org/html/2608.00860#bib.bib9); Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)\. The manifold hypothesis suggests that trained weights concentrate near a low\-dimensional set, possessing robust empirical\(Denilet al\.[2013](https://arxiv.org/html/2608.00860#bib.bib6); Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)and theoretical\(Feffermanet al\.[2013](https://arxiv.org/html/2608.00860#bib.bib10)\)backing\. This intrinsic dimension remains manageable even as models scale to billions of parameters\(Aghajanyanet al\.[2021](https://arxiv.org/html/2608.00860#bib.bib9)\)\. While prior research used dense random subspaces primarily as analytical measurement tools, we repurpose them into a highly compact storage container\. We achieve this by making the random projection reproducible from an integer seed\.

#### Reparameterization\-based compression\.

Our method relates to techniques storing models as a seed alongside a small set of trainable coefficients\. Random projections have a long history in machine learning\(Rahimi and Recht[2007](https://arxiv.org/html/2608.00860#bib.bib11)\), and recent methods apply them directly to network parameters\. NOLA\(Koohpayeganiet al\.[2024](https://arxiv.org/html/2608.00860#bib.bib12)\)expresses weights as linear combinations of pseudo\-random basis networks, while MCNC\(Thrashet al\.[2025](https://arxiv.org/html/2608.00860#bib.bib13)\)constrains parameters to frozen nonlinear manifolds\. Our parameterization differs critically and adapts from\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)\. It is strictly nonlinear, centers the map at the standard initialization to encode a structural displacement rather than an absolute location, and utilizes latent quantization as the primary compression lever\.

#### Weight\-space compression\.

Traditional compression explicitly targets trained weights\. Early methods\(LeCunet al\.[1989](https://arxiv.org/html/2608.00860#bib.bib14)\)led to unstructured pruning that removes weights by magnitude\(Hanet al\.[2015](https://arxiv.org/html/2608.00860#bib.bib1)\)or by identifying winning lottery tickets\(Frankle and Carbin[2019](https://arxiv.org/html/2608.00860#bib.bib2); Chenet al\.[2020](https://arxiv.org/html/2608.00860#bib.bib15)\)\. Structured pruning offers hardware\-friendly alternatives\(Wenet al\.[2016](https://arxiv.org/html/2608.00860#bib.bib16)\)\. Quantization reduces numerical precision through post\-training methods\(Nagelet al\.[2020](https://arxiv.org/html/2608.00860#bib.bib17); Yaoet al\.[2021](https://arxiv.org/html/2608.00860#bib.bib18)\)or straight\-through quantization\-aware training\(Jacobet al\.[2018](https://arxiv.org/html/2608.00860#bib.bib3); Bengioet al\.[2013](https://arxiv.org/html/2608.00860#bib.bib19); Courbariauxet al\.[2015](https://arxiv.org/html/2608.00860#bib.bib20)\)\. Other techniques include weight clustering\(Hanet al\.[2016](https://arxiv.org/html/2608.00860#bib.bib4)\), hashing\(Chenet al\.[2015](https://arxiv.org/html/2608.00860#bib.bib21)\), and knowledge distillation\(Hintonet al\.[2015](https://arxiv.org/html/2608.00860#bib.bib5); Gouet al\.[2021](https://arxiv.org/html/2608.00860#bib.bib22)\)\. Our reparameterization paradigm is fundamentally complementary because it never represents the full weight set during optimization\. Instead, standard compression techniques like quantization are shifted entirely to the compact latent space to amplify the overall size reduction\.

#### Parameter\-efficient adaptation and hypernetworks\.

Parameter\-efficient fine\-tuning adapts large models with minimal storage overhead\. Standard adapters insert small bottleneck layers\(Houlsbyet al\.[2019](https://arxiv.org/html/2608.00860#bib.bib24)\), while prompt tuning optimizes hidden states\(Li and Liang[2021](https://arxiv.org/html/2608.00860#bib.bib23)\)\. Methods like LoRA\(Huet al\.[2022](https://arxiv.org/html/2608.00860#bib.bib25)\), QLoRA\(Dettmerset al\.[2023](https://arxiv.org/html/2608.00860#bib.bib26)\), and GaLore\(Zhaoet al\.[2024](https://arxiv.org/html/2608.00860#bib.bib27)\)constrain updates or gradients to low\-rank subspaces\. VeRA\(Kopiczkoet al\.[2024](https://arxiv.org/html/2608.00860#bib.bib28)\)freezes random low\-rank matrices and trains only scaling vectors\. Hypernetworks\(Haet al\.[2017](https://arxiv.org/html/2608.00860#bib.bib29); Chauhanet al\.[2024](https://arxiv.org/html/2608.00860#bib.bib30)\)generate weights using a secondary network but do not prioritize minimal storage footprints\. While our framework focuses on from\-scratch training for extreme compression, it naturally accommodates adaptation\. Centering the mapping on pretrained weights turns our latent vector into a kilobyte\-sized adapter \(Section[4](https://arxiv.org/html/2608.00860#S4.SSx4)\), offering a hyper\-compressed counterpart\.

## 3Preliminaries: Mapping Networks

We follow the Mapping Network formulation introduced in\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)\. Letfθf\_\{\\theta\}be a target network with flattened trainable parametersθ∈ℝP\\theta\\in\\mathbb\{R\}^\{P\}, wherePPis the total number of such parameters, and writefθ​\(x\)f\_\{\\theta\}\(x\)for its output on inputxx\. Ordinarilyθ\\thetais optimized directly by minimizing a task lossℒ\\mathcal\{L\}over a dataset\. A Mapping Network instead introduces a trainable latent vector𝐳∈ℝd\\mathbf\{z\}\\in\\mathbb\{R\}^\{d\}withd≪Pd\\ll Pand a fixed, differentiable mapg:ℝd→ℝPg:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{P\}, and setsθ=g​\(𝐳\)\\theta=g\(\\mathbf\{z\}\)\. Concretely,

g​\(𝐳\)=σ​\(W0​𝐳\+α​‖𝐳‖22​1\+b0\),g\(\\mathbf\{z\}\)\\;=\\;\\sigma\\\!\\left\(W\_\{0\}\\,\\mathbf\{z\}\\;\+\\;\\alpha\\left\\lVert\\mathbf\{z\}\\right\\rVert\_\{2\}^\{2\}\\,\\mathbf\{1\}\\;\+\\;b\_\{0\}\\right\),\(1\)whereW0∈ℝP×dW\_\{0\}\\in\\mathbb\{R\}^\{P\\times d\}is a fixed projection with near\-orthonormal columns,b0∈ℝPb\_\{0\}\\in\\mathbb\{R\}^\{P\}is a fixed center,α∈ℝ\\alpha\\in\\mathbb\{R\}is a small scalar modulation coefficient,𝟏∈ℝP\\mathbf\{1\}\\in\\mathbb\{R\}^\{P\}is the all\-ones vector,σ\\sigmais an element\-wise nonlinearity taken to betanh\\tanh, and∥⋅∥2\\left\\lVert\\cdot\\right\\rVert\_\{2\}is the Euclidean norm\. The centerb0b\_\{0\}is set to the target network’s standard initialization, so that at𝐳=0\\mathbf\{z\}=0the generated parameters equal a well\-scaled initial network and the latent learns a displacement from it\. The termα​‖𝐳‖22\\alpha\\left\\lVert\\mathbf\{z\}\\right\\rVert\_\{2\}^\{2\}is a scalar, added to every coordinate, that follows from the additive weight modulation of the original formulation and is negligible for the smallα\\alphawe use\.

Training minimizes the task loss with respect to the latent only\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\),

𝐳⋆=arg⁡min𝐳∈ℝd⁡𝔼\(x,y\)​ℒ​\(fg​\(𝐳\)​\(x\),y\),\\mathbf\{z\}^\{\\star\}\\;=\\;\\arg\\min\_\{\\mathbf\{z\}\\in\\mathbb\{R\}^\{d\}\}\\;\\mathbb\{E\}\_\{\(x,y\)\}\\,\\mathcal\{L\}\\\!\\left\(f\_\{g\(\\mathbf\{z\}\)\}\(x\),\\,y\\right\),\(2\)withW0W\_\{0\},b0b\_\{0\}, andα\\alphaheld fixed, so that gradients flow throughgginto𝐳\\mathbf\{z\}but never intoW0W\_\{0\}orb0b\_\{0\}\. The target network is used only for its forward pass\. The Mapping Theorem of\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)shows, under Lipschitz and smoothness assumptions on the loss and the map, that for a target parameterθ⋆\\theta^\{\\star\}on a low\-dimensional manifold there exists a latent𝐳⋆\\mathbf\{z\}^\{\\star\}withg​\(𝐳⋆\)g\(\\mathbf\{z\}^\{\\star\}\)arbitrarily close toθ⋆\\theta^\{\\star\}, which is the existence statement that motivates optimizing in𝐳\\mathbf\{z\}\.

#### Two training regimes\.

\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)distinguish two ways of applying the Mapping Network to a target, which they call*Single Latent Vector Training*\(SLVT\) and*Layer\-Wise Training*\(LWT\)\. In SLVT, a single latent vector generates all of the target’s mapped parameters at once, exactly as in Equations \([1](https://arxiv.org/html/2608.00860#S3.E1)\) and \([2](https://arxiv.org/html/2608.00860#S3.E2)\)\. In LWT the target is instead handled one layer at a time, where a layer is a single trainable module of the network’s architecture \(an individual convolutional or fully\-connected layer\): the mapped parameters are partitioned by layer,θmap=\(θ1,…,θL\)\\theta\_\{\\mathrm\{map\}\}=\(\\theta^\{1\},\\dots,\\theta^\{L\}\), withθℓ\\theta^\{\\ell\}the parameters of theℓ\\ell\-th layer,PℓP\_\{\\ell\}their count, and∑ℓ=1LPℓ=P\\sum\_\{\\ell=1\}^\{L\}P\_\{\\ell\}=P, and each layer carries its own latent𝐳ℓ∈ℝdℓ\\mathbf\{z\}\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\}, frozen projectionW0ℓ∈ℝPℓ×dℓW\_\{0\}^\{\\ell\}\\in\\mathbb\{R\}^\{P\_\{\\ell\}\\times d\_\{\\ell\}\}, and centerb0ℓb\_\{0\}^\{\\ell\}:

θℓ=σ​\(W0ℓ​𝐳ℓ\+α​‖𝐳ℓ‖22​1\+b0ℓ\),ℓ=1,…,L\.\\theta^\{\\ell\}\\;=\\;\\sigma\\\!\\left\(W\_\{0\}^\{\\ell\}\\,\\mathbf\{z\}\_\{\\ell\}\\;\+\\;\\alpha\\left\\lVert\\mathbf\{z\}\_\{\\ell\}\\right\\rVert\_\{2\}^\{2\}\\,\\mathbf\{1\}\\;\+\\;b\_\{0\}^\{\\ell\}\\right\),\\qquad\\ell=1,\\dots,L\.\(3\)SLVT is the special case whereL=1L=1\. LWT spends a larger total latent budgetD=∑ℓdℓD=\\sum\_\{\\ell\}d\_\{\\ell\}in exchange for finer control, since the per\-layer dimensionsdℓd\_\{\\ell\}can be allocated in proportion to each layer’s size or difficulty rather than sharing one latent across the whole network\. Everything that follows, including the deployable artifact, the seeded basis, and latent quantization, applies unchanged to each per\-layer block\. To keep the notation light, we state it for a single latent and flag the layer\-wise sum where it matters\.\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)also applies the same map beyond from\-scratch training, to fine\-tune a pretrained network by additive modulation of its weights, and we build on this in Section[4](https://arxiv.org/html/2608.00860#S4.SSx4)\.

The property we exploit is structural rather than about accuracy\. In Eq\. \([1](https://arxiv.org/html/2608.00860#S3.E1)\), the objects that carry learned information and the objects that are fixed are cleanly separated: only𝐳\\mathbf\{z\}is trained, whileW0W\_\{0\},b0b\_\{0\}, andα\\alphaare fixed the moment they are chosen\. If those fixed objects can be regenerated on demand, the trained model is fully described by𝐳\\mathbf\{z\}\.

## 4Kilobyte Models

seedss\(8 bytes\)quantized latentQb​\(𝐳⋆\)Q\_\{b\}\(\\mathbf\{z\}^\{\\star\}\)\(d​b/8db/8bytes\)stored / transmitted artifactregenerateW0=ρ​\(s\),b0=ρ′​\(s\)W\_\{0\}=\\rho\(s\),\\ b\_\{0\}=\\rho^\{\\prime\}\(s\)θ=σ​\(W0​𝐳\+α​‖𝐳‖2​𝟏\+b0\)\\theta=\\sigma\(W\_\{0\}\\mathbf\{z\}\+\\alpha\\left\\lVert\\mathbf\{z\}\\right\\rVert^\{2\}\\mathbf\{1\}\+b\_\{0\}\)targetfθ​\(x\)f\_\{\\theta\}\(x\)Figure 1:The Kilobyte model artifact\. Only the seedssand the quantized latentQb​\(𝐳⋆\)Q\_\{b\}\(\\mathbf\{z\}^\{\\star\}\)are stored or transmitted \(boxed\)\. On the receiving device, the seed regenerates the frozen projectionW0W\_\{0\}and centerb0b\_\{0\}, the map of Eq\. \([1](https://arxiv.org/html/2608.00860#S3.E1)\) produces the full parameter vectorθ\\theta, and the target network runs a standard forward pass\. The artifact size is set by the latent dimensionddand bit widthbb, not by the parameter countPP\.### The deployable artifact

The parameters of a real network include normalization affine parameters \(the scale and shift of group, layer, or batch normalization\), whose standard initialization is the all\-ones and all\-zeros vectors\. These lie on the boundary of the range oftanh\\tanhand cannot be produced by Eq\. \([1](https://arxiv.org/html/2608.00860#S3.E1)\)\. Furthermore, they are also few in number\. We therefore partition the target parameters asθ=\(θmap,θnorm\)\\theta=\(\\theta\_\{\\mathrm\{map\}\},\\theta\_\{\\mathrm\{norm\}\}\), whereθmap∈ℝP\\theta\_\{\\mathrm\{map\}\}\\in\\mathbb\{R\}^\{P\}are the convolutional and linear weights and biases, which form the overwhelming majority and are generated by the map, andθnorm∈ℝPn\\theta\_\{\\mathrm\{norm\}\}\\in\\mathbb\{R\}^\{P\_\{n\}\}are the normalization parameters, withPn≪PP\_\{n\}\\ll P, which are trained directly and stored verbatim\.

A trained Kilobyte model is then the triple

𝒜=\(s,Qb​\(𝐳⋆\),θnorm\),\\mathcal\{A\}\\;=\\;\\bigl\(\\,s,\\;Q\_\{b\}\(\\mathbf\{z\}^\{\\star\}\),\\;\\theta\_\{\\mathrm\{norm\}\}\\,\\bigr\),\(4\)wheressis the integer seed that generatesW0W\_\{0\}andb0b\_\{0\},QbQ\_\{b\}is abb\-bit quantizer, and𝐳⋆\\mathbf\{z\}^\{\\star\}is the trained latent\. To deploy the model, a device regeneratesW0W\_\{0\}andb0b\_\{0\}fromss, dequantizes the latent, evaluates Eq\. \([1](https://arxiv.org/html/2608.00860#S3.E1)\) to obtainθmap\\theta\_\{\\mathrm\{map\}\}, reattachesθnorm\\theta\_\{\\mathrm\{norm\}\}, and runs the ordinary forward pass offθf\_\{\\theta\}\(Figure[1](https://arxiv.org/html/2608.00860#S4.F1)\)\. The size of the artifact in bytes is

\|𝒜\|=d​b8⏟latent payload\+4⏟scale\+8⏟seed\+4​Pn⏟norm params,\|\\mathcal\{A\}\|\\;=\\;\\underbrace\{\\frac\{d\\,b\}\{8\}\}\_\{\\text\{latent payload\}\}\\;\+\\;\\underbrace\{4\}\_\{\\text\{scale\}\}\\;\+\\;\\underbrace\{8\}\_\{\\text\{seed\}\}\\;\+\\;\\underbrace\{4P\_\{n\}\}\_\{\\text\{norm params\}\},\(5\)where the dominant term for the budgets we consider is the latent payloadd​b/8db/8\. The remaining terms are stored in full precision: each layer’s quantization scale is a single3232\-bit float, giving4​L4Lbytes across theLLlayers \(and just44bytes under SLVT, whereL=1L=1\); each of thePnP\_\{n\}directly\-trained normalization parameters is a3232\-bit float \(44bytes, hence4​Pn4P\_\{n\}\); and the seed is a single6464\-bit integer \(88bytes\), regardless of the number of layers, since one master seed generates everyW0ℓW\_\{0\}^\{\\ell\}\. Inference is unchanged relative to the target network, because after regeneration, the weights are dense floating\-point values \(the compression is of the stored and transmitted description, not of the compute\)\.

### Seeded basis generation at scale

For the artifact to remain small, the receiver must feasibly regenerateW0∈ℝP×dW\_\{0\}\\in\\mathbb\{R\}^\{P\\times d\}from the seed alone, even whenW0W\_\{0\}exceeds commodity accelerator memory \(e\.g\., tens of gigabytes for a 1\.2M parameter target withd=16,384d=16\{,\}384\)\. Unlike Mapping Networks\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\), we use a reproducible, near\-orthonormal projection generated in row blocks to avoid full matrix materialization\.

WritingW0=\[B1⊤,…,BK⊤\]⊤W\_\{0\}=\[B\_\{1\}^\{\\top\},\\dots,B\_\{K\}^\{\\top\}\]^\{\\top\}as a stack ofKKrow blocks, each blockBkB\_\{k\}is drawn from a pseudo\-random generator seeded by a mixing function of the seedssand block indexkk\. Entries are either Gaussian,\[Bk\]i​j∼𝒩​\(0,1/P\)\[B\_\{k\}\]\_\{ij\}\\sim\\mathcal\{N\}\(0,1/P\), or Rademacher,\[Bk\]i​j∈\{−1/P,\+1/P\}\[B\_\{k\}\]\_\{ij\}\\in\\\{\-1/\\sqrt\{P\},\+1/\\sqrt\{P\}\\\}\. These i\.i\.d\. entries allow independent block regeneration from\(s,k\)\(s,k\), avoiding the full materialization required by exact orthogonalization\. The1/P1/\\sqrt\{P\}scaling ensures expected near\-orthonormality,𝔼​\[W0⊤​W0\]=Id\\mathbb\{E\}\[W\_\{0\}^\{\\top\}W\_\{0\}\]=I\_\{d\}, meaning the map𝐳↦W0​𝐳\\mathbf\{z\}\\mapsto W\_\{0\}\\mathbf\{z\}approximately preserves inner products for largePP\.

This deterministic generation guarantees cross\-device portability\. For LWT, a single master seed incorporates the layer indexℓ\\ellto generate allLLprojectionsW0ℓW\_\{0\}^\{\\ell\}\. Memory permitting,W0W\_\{0\}can be cached, streamed, or regenerated dynamically using a custom block\-wise differentiable operator to compute𝐳↦W0​𝐳\\mathbf\{z\}\\mapsto W\_\{0\}\\mathbf\{z\}and its transpose\.

### Latent quantization

The latent payload shrinks directly with the bit widthbb, so we store𝐳\\mathbf\{z\}with standard symmetric uniform quantization\(Jacobet al\.[2018](https://arxiv.org/html/2608.00860#bib.bib3)\)and consider two variants\. Post\-training quantization \(PTQ\) simply rounds the already\-trained latent tobbbits\. Quantization\-aware training \(QAT\) instead fine\-tunes the latent with the rounding simulated in the loop, so its values settle where they survive quantization as gradients are passed through the rounding with a straight\-through estimator\(Bengioet al\.[2013](https://arxiv.org/html/2608.00860#bib.bib19)\)\. Because the latent has onlyddentries, this is inexpensive and touches a single small vector rather than the full weight set\. Algorithms for encoding and decoding are provided in the Appendix\.

### Fine\-tuning as a kilobyte delta

While the previous sections describe training from scratch, our framework naturally extends to fine\-tuning, similar to\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)\. By replacing the initialization centerb0b\_\{0\}in Eq\. \([1](https://arxiv.org/html/2608.00860#S3.E1)\) with a frozen pretrained parameter vectorθpre\\theta\_\{\\mathrm\{pre\}\}and using an identity activation, the map generates a displacement from the pretrained solution:

θft=θpre\+W0​𝐳,\\theta\_\{\\mathrm\{ft\}\}\\;=\\;\\theta\_\{\\mathrm\{pre\}\}\\;\+\\;W\_\{0\}\\,\\mathbf\{z\},\(6\)Here, only𝐳\\mathbf\{z\}is trained\. The latent encodes a task\-specific*adapter*within the frozen random subspace, similar to LoRA\(Huet al\.[2022](https://arxiv.org/html/2608.00860#bib.bib25)\)and VeRA\(Kopiczkoet al\.[2024](https://arxiv.org/html/2608.00860#bib.bib28)\), but reduced strictly to a seed and a quantized latent\. The seeded basis generation, quantization strategies, and training regimes apply unchanged\.

The byte accounting, however, must adapt\. Becauseθpre\\theta\_\{\\mathrm\{pre\}\}is a specific pretrained model rather than a seed\-regenerable initialization, it is treated as a shared prerequisite assumed to be already present on the deployment device\. A fine\-tuned Kilobyte model is therefore defined as:

𝒜ft=\(h,s,\{Qb​\(𝐳ℓ⋆\)\}ℓ=1L,θnorm\),\\mathcal\{A\}\_\{\\mathrm\{ft\}\}\\;=\\;\\bigl\(\\,h,\\;s,\\;\\\{Q\_\{b\}\(\\mathbf\{z\}\_\{\\ell\}^\{\\star\}\)\\\}\_\{\\ell=1\}^\{L\},\\;\\theta\_\{\\mathrm\{norm\}\}\\,\\bigr\),\(7\)wherehhis a short identifier \(hash\) of the base model\. This hash binds the delta to its training base, ensuring it is applied to the correct pretrained weights\.

## 5Experiments

We evaluate all targets under both SLVT and LWT regimes \(Section[3](https://arxiv.org/html/2608.00860#S3)\) using identical byte accounting\. We report results for both from\-scratch targets and a fine\-tuning adapter \(Section[4](https://arxiv.org/html/2608.00860#S4.SSx4)\) evaluated on a pretrained ResNet\-50\. Mapped and baseline models are trained via Adam\(Kingma and Ba[2014](https://arxiv.org/html/2608.00860#bib.bib31)\)\. Experiments are implemented in PyTorch with CUDA on an NVIDIA RTX 4060 GPU across 3 random seeds\. We report mean metrics; corresponding standard deviations and hyperparameters are detailed in the Appendix\.

### Seeded basis is reproducible and near\-orthonormal

Before evaluating accuracy we verify that the seeded basis of Section[4](https://arxiv.org/html/2608.00860#S4.SSx2)behaves as the parameterization requires\. Table[1](https://arxiv.org/html/2608.00860#S5.T1)summarizes the Gram matrixW0⊤​W0W\_\{0\}^\{\\top\}W\_\{0\}across the parameter counts of our targets: the columns have unit norm to within0\.1%0\.1\\%, and the off\-diagonal correlations are small and shrink as1/P1/\\sqrt\{P\}, so larger targets are closer to orthonormal\. The Gaussian and Rademacher constructions agree at everyPP, so this does not depend on the particular random basis\. We also verify the generator\. The cached and regenerated back\-ends produce identical projections, a fixed seed reproduces the basis exactly, and a device\-independent generator is bit\-identical across runs, in all three cases to a maximum difference of0\. The block\-wise operator agrees with the closed\-form transpose to2\.6×10−62\.6\\times 10^\{\-6\}\. A seed is therefore a portable description of the model\. For why the projection is taken to be orthogonal in the first place, please refer to\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)\.

Table 1:Near\-orthonormality of the seeded basis: Gram statistics ofW0⊤​W0W\_\{0\}^\{\\top\}W\_\{0\}over a sample of256256columns, for the Gaussian and Rademacher constructions at three parameter countsPP\.
### CNN Results

We evaluate the Kilobyte model artifact in the same classification setting of\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\): two convolutional targets, CNN1 \(538,081538\{,\}081parameters\) and CNN2 \(105,866105\{,\}866\), on MNIST\(Lecunet al\.[1998](https://arxiv.org/html/2608.00860#bib.bib33)\)and FashionMNIST\(Xiaoet al\.[2017](https://arxiv.org/html/2608.00860#bib.bib34)\)\. Both are plain convolutional networks with no normalization layers, so the artifact stores no separate normalization parameters\. We sweep the latent budgetd∈\{1024,2048,4096\}d\\in\\\{1024,2048,4096\\\}at full precision and at eight and four bits under both PTQ and QAT, and both training regimes: SLVT, and LWT withddsplit across the four conv/linear layers in proportion to their size \(which leaves the two regimes essentially the same artifact size at a givendd\)\. Tables[2](https://arxiv.org/html/2608.00860#S5.T2)and[3](https://arxiv.org/html/2608.00860#S5.T3)and Figure[2](https://arxiv.org/html/2608.00860#S5.F2)report the results\.

Table 2:CNN Results: Test accuracy \(%\) for the two targets trained conventionally \(“full”\) and as mapped Kilobyte models under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT, on MNIST and FashionMNIST\.Table 3:CNN artifact sizes \(bytes\)\. Mapped sizes depend only onddand are identical across CNN1/CNN2 and both datasets, and LWT adds1212bytes \(one extra scale per layer\)\.![Refer to caption](https://arxiv.org/html/2608.00860v1/x1.png)Figure 2:Accuracy against artifact size for the CNN targets\.The results show several key patterns\. First, QAT performs much better than PTQ at low bit widths\. For example, CNN1 on MNIST atd=4096d=4096improves from 96\.2% \(PTQ\) to 98\.5% \(QAT\) at four bits\. Second, eight\-bit quantization loses almost no accuracy compared to full precision\. Third, mapped models achieve high accuracy with a tiny footprint\. CNN2 reaches 98\.60% on MNIST using just 2,060 bytes at four\-bit QAT\. Fourth, mapped models trail full networks slightly on FashionMNIST due to latent capacity limits, but larger latent dimensions help close this gap\. Finally, SLVT and LWT methods perform similarly\. SLVT has a small advantage at larger latent sizes and holds the best four\-bit QAT scores\. However, LWT is better for low\-precision PTQ because it scales each layer separately\. This makes LWT the better option when fine\-tuning is not possible\.

### LSTM Results

We repeat the study on a recurrent target and a regression task, following the time\-series experiment of\(Sen and Mukherjee[2026](https://arxiv.org/html/2608.00860#bib.bib8)\)\. The target is a single\-layer LSTM with12,94912\{,\}949parameters trained on the Beijing air\-quality dataset\(Chen[2017](https://arxiv.org/html/2608.00860#bib.bib36)\)\. The metric is test Mean Squared Error \(MSE\), where lower is better\. As before, we sweep the latent dimensiond∈\{256,1024,2048\}d\\in\\\{256,1024,2048\\\}and the bit width under both quantizers, against the conventionally trained LSTM compressed by post\-training weight quantization\. Tables[4](https://arxiv.org/html/2608.00860#S5.T4)and[5](https://arxiv.org/html/2608.00860#S5.T5)and Figure[3](https://arxiv.org/html/2608.00860#S5.F3)report the results\.

![Refer to caption](https://arxiv.org/html/2608.00860v1/x2.png)Figure 3:Test MSE against artifact size on the air\-quality task \(both axes log, lower MSE is better\)The regression results match the classification patterns\. Mapped models achieve similar error rates to the full LSTM but at a much smaller size\. For instance, a four\-bit QAT model atd=256d=256gets an MSE of 0\.00061 using only 140 bytes\. This is about 370 times smaller than the full\-precision LSTM\. As seen before, QAT is much better than PTQ for lower bit widths, while eight\-bit quantization causes no accuracy loss\.

Table 4:LSTM Results on the Beijing air\-quality dataset: test MSE \(×10−3\\times 10^\{\-3\}\) for the LSTM trained conventionally \(“full”\) and as a mapped Kilobyte model under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT\.Table 5:LSTM artifact sizes \(bytes\)\. Mapped sizes depend only ondd, and LWT adds88bytes \(one extra scale per layer\)\.
### MLP Results

We evaluate fully\-connected targets on tabular data\. We use two standard tabular benchmarks, the binary particle\-physics dataset HIGGS\(Whiteson[2014](https://arxiv.org/html/2608.00860#bib.bib37)\)\(28 features, and we use a600600k\-row subset\) and the seven\-class forest Covertype dataset\(Blackard[1998](https://arxiv.org/html/2608.00860#bib.bib40)\)\(54 features\), and two three\-hidden\-layer MLPs: MLP1 with about five million parameters and MLP2 with about one million\. Because a denseP×dP\\times dprojection for a five\-million\-parameter target is tens of gigabytes, here we generate the weights with the structured implicit basis of Section[4](https://arxiv.org/html/2608.00860#S4.SSx2)\. The sweep and the metric \(test accuracy\) are otherwise identical to the CNN study\. Tables[6](https://arxiv.org/html/2608.00860#S5.T6)and[7](https://arxiv.org/html/2608.00860#S5.T7)and Figure[4](https://arxiv.org/html/2608.00860#S5.F4)report the results\.

Table 6:MLP Results on two tabular datasets: test accuracy \(%\) for MLP1 and MLP2 trained conventionally \(“full”\) and as mapped Kilobyte models under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT\.The MLP results show consistent compression benefits\. On the HIGGS dataset, mapped models closely match the full network’s accuracy using only a fraction of the memory\. On Covertype, the mapped models fall slightly behind the full network but improve quickly as the latent dimension increases\. This shows that any accuracy gap is due to limited latent capacity\. Once again, QAT is crucial for low bit widths, while eight\-bit quantization incurs minimal performance drop\. LWT and SLVT perform similarly on these tabular tasks, though SLVT keeps a slight advantage\.

![Refer to caption](https://arxiv.org/html/2608.00860v1/x3.png)Figure 4:Accuracy against artifact size for the MLP targets\.Table 7:MLP artifact sizes \(bytes\)\. Mapped \(SLVT\) sizes depend only onddand are identical across models and datasets\.
### Fine\-tuning Results

We now evaluate the fine\-tuning regime of Section[4](https://arxiv.org/html/2608.00860#S4.SSx4), where the Kilobyte model is a task adapter over a pretrained base\. The base is an ImageNet\-pretrained ResNet\-50\(Heet al\.[2016](https://arxiv.org/html/2608.00860#bib.bib38)\)\(23,583,84523\{,\}583\{,\}845parameters\) with a fresh3737\-way head, adapted to the Oxford\-IIIT Pet dataset\(Parkhiet al\.[2012](https://arxiv.org/html/2608.00860#bib.bib39)\)\(3,6803\{,\}680training and3,6693\{,\}669test images at224×224224\{\\times\}224\) by training only the mapped delta of Eq\. \([6](https://arxiv.org/html/2608.00860#S4.E6)\)\. BatchNorm is frozen as part of the referenced base, so the artifact is just the seed and quantized latent, no verbatim weights\. As for the MLP, the megaparameter target uses the structured basis\. We sweepd∈\{2048,8192,16384\}d\\in\\\{2048,8192,16384\\\}under both regimes and quantizers \(fp32/int8/int4\), training the latent with gradient clipping, which is needed for stability at the largest budget\. The baseline is full fine\-tuning of all weights, compressed post\-training\. Tables[8](https://arxiv.org/html/2608.00860#S5.T8)and[9](https://arxiv.org/html/2608.00860#S5.T9)and Figure[5](https://arxiv.org/html/2608.00860#S5.F5)report the results\.

![Refer to caption](https://arxiv.org/html/2608.00860v1/x4.png)Figure 5:Pets accuracy against artifact size for the fine\-tuned adapter, SLVT and LWT, against the full fine\-tuned ResNet\-50 compressed by weight quantization\.Table 8:Fine\-tuning Results: Pets test accuracy \(%\) for an ImageNet\-pretrained ResNet\-50 adapted by a mapped delta under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT\. The baseline is full fine\-tuning\.Table 9:Fine\-tuning artifact sizes \(bytes\)\.The adapter acts as a highly effective fine\-tuning tool\. It comes within three accuracy points of full fine\-tuning while being thousands of times smaller\. For example, a four\-bit QAT adapter uses just 4,116 bytes to reach 88\.4% accuracy, compared to a 94\-megabyte full network\. Unlike the full ResNet\-50, which drops to 2\.9% accuracy under four\-bit PTQ, the mapped adapter easily survives aggressive quantization\. In this setting, SLVT performs slightly better than LWT\. LWT allocates capacity based on layer size, which starves the tiny but crucial task head\. SLVT avoids this by letting the head use the entire shared budget\.

#### Comparison to LoRA and VeRA\.

We compare our approach to parameter\-efficient fine\-tuning methods like LoRA\(Huet al\.[2022](https://arxiv.org/html/2608.00860#bib.bib25)\)and VeRA\(Kopiczkoet al\.[2024](https://arxiv.org/html/2608.00860#bib.bib28)\)\. Both baselines adapt the same weight matrices and are evaluated across different ranks using PTQ, while our adapter uses QAT\. Results are in Table[10](https://arxiv.org/html/2608.00860#S5.T10)and Figure[6](https://arxiv.org/html/2608.00860#S5.F6)\.

Table 10:Adapter comparison on ResNet\-50→\\toOxford\-IIIT Pets: trainable parameters and accuracy with artifact size at full precision and at four bits\.Our adapter, at four bits, reaches 88\.4% accuracy using just 4 KB, outperforming the smallest VeRA model, which needs 13\.9 KB for 87\.6% accuracy\. While LoRA and VeRA achieve higher accuracy given larger storage budgets \(up to 92\.7% at 0\.6 MB\), our single latent plateaus near 89% due to capacity limits\. However, our low\-dimensional latent survives quantization much better than full weights\. For instance, a four\-bit adapter maintains nearly 88% accuracy, whereas a four\-bit full network collapses to 2\.9%\. Ultimately, our mapped adapter is ideal for strict kilobyte limits, offering a 5 to 160 times smaller artifact at a slight accuracy cost, while LoRA and VeRA are better when larger sizes are acceptable\.

![Refer to caption](https://arxiv.org/html/2608.00860v1/x5.png)Figure 6:Accuracy against artifact size for the fine\-tuning adapters\.

## 6Conclusion

We introduced Kilobyte models, an extreme compression paradigm that stores neural networks as a deterministic seed and a quantized latent vector rather than as a collection of weights\. By leveraging a reproducible, block\-wise random basis, we successfully decouple the deployable artifact size from the target network’s actual parameter count\. Our experiments demonstrate that this formulation, particularly when paired with quantization\-aware training, matches the accuracy of aggressively quantized full networks at a fraction of the storage cost\. From achieving 98\.6% accuracy on MNIST in just 2 KB to fine\-tuning a ResNet\-50 adapter in 4 KB, Kilobyte models prove highly effective in the ultra\-low storage regime\. Furthermore, our approach outperforms existing parameter\-efficient methods, such as LoRA and VeRA, when constrained to strict kilobyte budgets\. Ultimately, this framework demonstrates that trading explicit parameter storage for latent capacity is a powerful strategy for bandwidth\-constrained deployments\.

#### Limitations\.

Kilobyte models drastically reduce storage but do not compress inference compute, as regenerating dense weights leaves inference latency and memory unchanged\. Second, accuracy is bottlenecked by latent capacity, causing slight performance drops on harder datasets like FashionMNIST and Covertype\. Third, fine\-tuning adapters requires the prerequisite base model to be locally present on the deployment device\. Finally, Mapping Networks are not yet extended to Transformers and Large Language Models, meaning Kilobyte models cannot currently support them\. We are actively working on extending both frameworks to these architectures\.

## References

- A\. Aghajanyan, S\. Gupta, and L\. Zettlemoyer \(2021\)Intrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 7319–7328\.External Links:[Link](https://aclanthology.org/2021.acl-long.568/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.568)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Bengio, N\. Léonard, and A\. Courville \(2013\)Estimating or propagating gradients through stochastic neurons for conditional computation\.External Links:1308\.3432,[Link](https://arxiv.org/abs/1308.3432)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.00860#S4.SSx3.p1.4)\.
- J\. Blackard \(1998\)Covertype\.Note:UCI Machine Learning RepositoryDOI: https://doi\.org/10\.24432/C50K5NCited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx4.p1.2)\.
- V\. K\. Chauhan, J\. Zhou, P\. Lu, S\. Molaei, and D\. A\. Clifton \(2024\)A brief review of hypernetworks in deep learning\.Artificial Intelligence Review57\(9\)\.External Links:ISSN 1573\-7462,[Link](http://dx.doi.org/10.1007/s10462-024-10862-8),[Document](https://dx.doi.org/10.1007/s10462-024-10862-8)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Chen \(2017\)Beijing Multi\-Site Air Quality\.Note:UCI Machine Learning RepositoryDOI: https://doi\.org/10\.24432/C5RK5GCited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx3.p1.2)\.
- T\. Chen, J\. Frankle, S\. Chang, S\. Liu, Y\. Zhang, Z\. Wang, and M\. Carbin \(2020\)The lottery ticket hypothesis for pre\-trained bert networks\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20,Red Hook, NY, USA\.External Links:ISBN 9781713829546Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Chen, J\. T\. Wilson, S\. Tyree, K\. Q\. Weinberger, and Y\. Chen \(2015\)Compressing neural networks with the hashing trick\.pp\. 2285–2294\.Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Courbariaux, Y\. Bengio, and J\. David \(2015\)BinaryConnect: training deep neural networks with binary weights during propagations\.Cambridge, MA, USA,pp\. 3123–3131\.Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Denil, B\. Shakibi, L\. Dinh, M\. Ranzato, and N\. de Freitas \(2013\)Predicting parameters in deep learning\.InProceedings of the 27th International Conference on Neural Information Processing Systems \- Volume 2,NIPS’13,Red Hook, NY, USA,pp\. 2148–2156\.Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p2.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized llms\.pp\. 10088–10115\.External Links:[Document](https://dx.doi.org/10.52202/075280-0441),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/1feb87871436031bdc0f2beaa62a049b-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1)\.
- C\. Fefferman, S\. Mitter, and H\. Narayanan \(2013\)Testing the manifold hypothesis\.External Links:1310\.0425,[Link](https://arxiv.org/abs/1310.0425)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Frankle and M\. Carbin \(2019\)The lottery ticket hypothesis: finding sparse, trainable neural networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJl-b3RcF7)Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p1.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Gou, B\. Yu, S\. J\. Maybank, and D\. Tao \(2021\)Knowledge distillation: a survey\.Int\. J\. Comput\. Vision129\(6\),pp\. 1789–1819\.External Links:ISSN 0920\-5691,[Link](https://doi.org/10.1007/s11263-021-01453-z),[Document](https://dx.doi.org/10.1007/s11263-021-01453-z)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Ha, A\. M\. Dai, and Q\. V\. Le \(2017\)HyperNetworks\.External Links:[Link](https://openreview.net/forum?id=rkpACe1lx)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Han, H\. Mao, and W\. J\. Dally \(2016\)Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding\.External Links:1510\.00149,[Link](https://arxiv.org/abs/1510.00149)Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p1.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Han, J\. Pool, J\. Tran, and W\. J\. Dally \(2015\)Learning both weights and connections for efficient neural networks\.InProceedings of the 29th International Conference on Neural Information Processing Systems \- Volume 1,NIPS’15,Cambridge, MA, USA,pp\. 1135–1143\.Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p1.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.pp\. 770–778\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.90)Cited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx5.p1.6)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p1.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly \(2019\)Parameter\-efficient transfer learning for NLP\.pp\. 2790–2799\.External Links:[Link](https://proceedings.mlr.press/v97/houlsby19a.html)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.00860#S4.SSx4.p1.3),[§5](https://arxiv.org/html/2608.00860#S5.SSx5.SSS0.Px1.p1.1)\.
- B\. Jacob, S\. Kligys, B\. Chen, M\. Zhu, M\. Tang, A\. Howard, H\. Adam, and D\. Kalenichenko \(2018\)Quantization and training of neural networks for efficient integer\-arithmetic\-only inference\.In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,Vol\.,pp\. 2704–2713\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2018.00286)Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p1.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.00860#S4.SSx3.p1.4)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.CoRRabs/1412\.6980\.External Links:[Link](https://api.semanticscholar.org/CorpusID:6628106)Cited by:[§5](https://arxiv.org/html/2608.00860#S5.p1.1)\.
- S\. A\. Koohpayegani, N\. K\. L, P\. Nooralinejad, S\. Kolouri, and H\. Pirsiavash \(2024\)NOLA: compressing loRA using linear combination of random basis\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TjfXcDgvzk)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Kopiczko, T\. Blankevoort, and Y\. Asano \(2024\)VeRA: vector\-based random matrix adaptation\.pp\. 6815–6835\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/1b53ad08de383a049e9668a9d0b6a053-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.00860#S4.SSx4.p1.3),[§5](https://arxiv.org/html/2608.00860#S5.SSx5.SSS0.Px1.p1.1)\.
- Y\. Lecun, L\. Bottou, Y\. Bengio, and P\. Haffner \(1998\)Gradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.External Links:[Document](https://dx.doi.org/10.1109/5.726791)Cited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx2.p1.5)\.
- Y\. LeCun, J\. Denker, and S\. Solla \(1989\)Optimal brain damage\.InAdvances in Neural Information Processing Systems,D\. Touretzky \(Ed\.\),Vol\.2,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Li, H\. Farkhoor, R\. Liu, and J\. Yosinski \(2018\)Measuring the intrinsic dimension of objective landscapes\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ryup8-WCW)Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p2.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px1.p1.1)\.
- X\. L\. Li and P\. Liang \(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.Online,pp\. 4582–4597\.External Links:[Link](https://aclanthology.org/2021.acl-long.353/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.353)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Nagel, R\. A\. Amjad, M\. Van Baalen, C\. Louizos, and T\. Blankevoort \(2020\)Up or down? adaptive rounding for post\-training quantization\.InProceedings of the 37th International Conference on Machine Learning,ICML’20\.Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- O\. M\. Parkhi, A\. Vedaldi, A\. Zisserman, and C\. V\. Jawahar \(2012\)Cats and dogs\.Cited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx5.p1.6)\.
- A\. Rahimi and B\. Recht \(2007\)Random features for large\-scale kernel machines\.InAdvances in Neural Information Processing Systems,J\. Platt, D\. Koller, Y\. Singer, and S\. Roweis \(Eds\.\),Vol\.20,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Sen and S\. Mukherjee \(2026\)Mapping networks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2608.00860#S1.p3.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.00860#S3.SS0.SSS0.Px1.p1.11),[§3](https://arxiv.org/html/2608.00860#S3.SS0.SSS0.Px1.p1.8),[§3](https://arxiv.org/html/2608.00860#S3.p1.11),[§3](https://arxiv.org/html/2608.00860#S3.p2.12),[§3](https://arxiv.org/html/2608.00860#S3.p2.13),[§4](https://arxiv.org/html/2608.00860#S4.SSx2.p1.3),[§4](https://arxiv.org/html/2608.00860#S4.SSx4.p1.2),[§5](https://arxiv.org/html/2608.00860#S5.SSx1.p1.6),[§5](https://arxiv.org/html/2608.00860#S5.SSx2.p1.5),[§5](https://arxiv.org/html/2608.00860#S5.SSx3.p1.2)\.
- C\. Thrash, R\. Andreas, A\. Abbasi, P\. Nooralinejad, S\. A\. Koohpayegani, H\. Pirsiavash, and S\. Kolouri \(2025\)MCNC: manifold\-constrained reparameterization for neural compression\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VMV8gefvq8)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Wen, C\. Wu, Y\. Wang, Y\. Chen, and H\. Li \(2016\)Learning structured sparsity in deep neural networks\.InAdvances in Neural Information Processing Systems,D\. Lee, M\. Sugiyama, U\. Luxburg, I\. Guyon, and R\. Garnett \(Eds\.\),Vol\.29,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2016/file/41bfd20a38bb1b0bec75acf0845530a7-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Whiteson \(2014\)HIGGS\.Note:UCI Machine Learning RepositoryDOI: https://doi\.org/10\.24432/C5V312Cited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx4.p1.2)\.
- H\. Xiao, K\. Rasul, and R\. Vollgraf \(2017\)Fashion\-mnist: a novel image dataset for benchmarking machine learning algorithms\.External Links:cs\.LG/1708\.07747Cited by:[§5](https://arxiv.org/html/2608.00860#S5.SSx2.p1.5)\.
- Z\. Yao, Z\. Dong, Z\. Zheng, A\. Gholami, J\. Yu, E\. Tan, L\. Wang, Q\. Huang, Y\. Wang, M\. Mahoney, and K\. Keutzer \(2021\)HAWQ\-v3: dyadic neural network quantization\.InProceedings of the 38th International Conference on Machine LearningProceedings of the 29th International Conference on Neural Information Processing Systems \- Volume 2Proceedings of the 32nd International Conference on Machine Learning \- Volume 37Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)Proceedings of the 36th International Conference on Machine LearningInternational Conference on Learning RepresentationsAdvances in Neural Information Processing SystemsProceedings of the 41st International Conference on Machine LearningInternational Conference on Learning RepresentationsInternational Conference on Learning RepresentationsComputer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8\-14, 2018, Proceedings, Part XIII2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)IEEE Conference on Computer Vision and Pattern Recognition,M\. Meila, T\. Zhang, C\. Zong, F\. Xia, W\. Li, R\. Navigli, K\. Chaudhuri, R\. Salakhutdinov, A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, S\. Levine, B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Proceedings of Machine Learning ResearchNIPS’15ICML’15Proceedings of Machine Learning ResearchICML’24, Vol\.13997362024,pp\. 11875–11886\.External Links:[Link](https://proceedings.mlr.press/v139/yao21a.html)Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)GaLore: memory\-efficient llm training by gradient low\-rank projection\.Cited by:[§2](https://arxiv.org/html/2608.00860#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AAppendix: Kilobyte models pseudocode

Algorithm 1Kilobyte model: encoding \(training and quantization\)1:target architecture

ff, data distribution

𝒟\\mathcal\{D\}, task loss

ℒ\\mathcal\{L\}, layer partition

LLwith latent budgets

\{dℓ\}ℓ=1L\\\{d\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}, bit width

bb, activation

σ\\sigma, modulation coefficient

α\\alpha, integer seed

ss
2:artifact

𝒜\\mathcal\{A\}
3:partition the trainable parameters of

ffinto mapped

θmap=\(θ1,…,θL\)\\theta\_\{\\mathrm\{map\}\}=\(\\theta^\{1\},\\dots,\\theta^\{L\}\), with

θℓ∈ℝPℓ\\theta^\{\\ell\}\\in\\mathbb\{R\}^\{P\_\{\\ell\}\}, and unmapped

θnorm∈ℝPn\\theta\_\{\\mathrm\{norm\}\}\\in\\mathbb\{R\}^\{P\_\{n\}\}⊳\\trianglerightL=1L=1is SLVT;L\>1L\>1is LWT

4:for

ℓ=1\\ell=1to

LLdo

5:

\(W0ℓ,b0ℓ\)←ρ​\(s,ℓ\)\(W\_\{0\}^\{\\ell\},\\,b\_\{0\}^\{\\ell\}\)\\leftarrow\\rho\(s,\\ell\)⊳\\trianglerightgenerated from the seed, never stored

6:endfor

7:

𝐳ℓ←ϵ​ξℓ\\mathbf\{z\}\_\{\\ell\}\\leftarrow\\epsilon\\,\\xi\_\{\\ell\}with

ξℓ∼𝒩​\(0,Idℓ\)\\xi\_\{\\ell\}\\sim\\mathcal\{N\}\(0,I\_\{d\_\{\\ell\}\}\)for all

ℓ\\ell;

θnorm←\\theta\_\{\\mathrm\{norm\}\}\\leftarrowstandard initialization

8:Latent training: only𝐳\\mathbf\{z\}andθnorm\\theta\_\{\\mathrm\{norm\}\}receive gradients\.

9:repeat

10:draw a minibatch

\(x,y\)∼𝒟\(x,y\)\\sim\\mathcal\{D\}
11:for

ℓ=1\\ell=1to

LLdo

12:

θℓ←σ​\(W0ℓ​𝐳ℓ\+α​‖𝐳ℓ‖22​𝟏\+b0ℓ\)\\theta^\{\\ell\}\\leftarrow\\sigma\\\!\\left\(W\_\{0\}^\{\\ell\}\\mathbf\{z\}\_\{\\ell\}\+\\alpha\\left\\lVert\\mathbf\{z\}\_\{\\ell\}\\right\\rVert\_\{2\}^\{2\}\\mathbf\{1\}\+b\_\{0\}^\{\\ell\}\\right\)
13:endfor

14:

J←ℒ​\(f\(θmap,θnorm\)​\(x\),y\)J\\leftarrow\\mathcal\{L\}\\\!\\left\(f\_\{\(\\theta\_\{\\mathrm\{map\}\},\\,\\theta\_\{\\mathrm\{norm\}\}\)\}\(x\),\\,y\\right\)
15:update

\{𝐳ℓ\}\\\{\\mathbf\{z\}\_\{\\ell\}\\\}and

θnorm\\theta\_\{\\mathrm\{norm\}\}by gradient descent on

JJ⊳\\trianglerightW0ℓ,b0ℓ,αW\_\{0\}^\{\\ell\},b\_\{0\}^\{\\ell\},\\alphafrozen

16:untilconverged, giving

\{𝐳ℓ⋆\}\\\{\\mathbf\{z\}\_\{\\ell\}^\{\\star\}\\\}
17:Quantization tobbbits, one scale per layer\.

18:ifpost\-training quantizationthen

19:

𝐳^ℓ←Qb​\(𝐳ℓ⋆\)\\hat\{\\mathbf\{z\}\}\_\{\\ell\}\\leftarrow Q\_\{b\}\(\\mathbf\{z\}\_\{\\ell\}^\{\\star\}\)for all

ℓ\\ell
20:else⊳\\trianglerightquantization\-aware training

21:repeat

22:draw

\(x,y\)∼𝒟\(x,y\)\\sim\\mathcal\{D\};

𝐳~ℓ←𝐳ℓ\+\(Qb​\(𝐳ℓ\)−𝐳ℓ\)detach\\tilde\{\\mathbf\{z\}\}\_\{\\ell\}\\leftarrow\\mathbf\{z\}\_\{\\ell\}\+\\bigl\(Q\_\{b\}\(\\mathbf\{z\}\_\{\\ell\}\)\-\\mathbf\{z\}\_\{\\ell\}\\bigr\)\_\{\\mathrm\{detach\}\}⊳\\trianglerightstraight\-through estimator

23:evaluate

JJas above using

𝐳~ℓ\\tilde\{\\mathbf\{z\}\}\_\{\\ell\}and update

\{𝐳ℓ\}\\\{\\mathbf\{z\}\_\{\\ell\}\\\}
24:untilconverged

25:

𝐳^ℓ←Qb​\(𝐳ℓ\)\\hat\{\\mathbf\{z\}\}\_\{\\ell\}\\leftarrow Q\_\{b\}\(\\mathbf\{z\}\_\{\\ell\}\)for all

ℓ\\ell
26:endif

27:return

𝒜=\(s,\{𝐳^ℓ\}ℓ=1L,θnorm\)\\mathcal\{A\}=\\bigl\(s,\\;\\\{\\hat\{\\mathbf\{z\}\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\},\\;\\theta\_\{\\mathrm\{norm\}\}\\bigr\), of size

\|𝒜\|=D​b/8\+4​L\+8\+4​Pn\|\\mathcal\{A\}\|=Db/8\+4L\+8\+4P\_\{n\}bytes, where

D=∑ℓdℓD=\\sum\_\{\\ell\}d\_\{\\ell\}

Algorithm 2Kilobyte model: decoding \(deployment on the receiving device\)1:artifact

𝒜=\(s,\{𝐳^ℓ\}ℓ=1L,θnorm\)\\mathcal\{A\}=\\bigl\(s,\\\{\\hat\{\\mathbf\{z\}\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\},\\theta\_\{\\mathrm\{norm\}\}\\bigr\), target architecture

ff, activation

σ\\sigma, modulation coefficient

α\\alpha
2:a ready\-to\-run network

fθf\_\{\\theta\}
3:for

ℓ=1\\ell=1to

LLdo

4:

\(W0ℓ,b0ℓ\)←ρ​\(s,ℓ\)\(W\_\{0\}^\{\\ell\},\\,b\_\{0\}^\{\\ell\}\)\\leftarrow\\rho\(s,\\ell\)⊳\\trianglerightsame seed reproduces the same objects

5:

𝐳ℓ←Δℓ⋅int​\(𝐳^ℓ\)\\mathbf\{z\}\_\{\\ell\}\\leftarrow\\Delta\_\{\\ell\}\\cdot\\mathrm\{int\}\(\\hat\{\\mathbf\{z\}\}\_\{\\ell\}\)⊳\\trianglerightdequantize with the stored per\-layer scale

6:

θℓ←σ​\(W0ℓ​𝐳ℓ\+α​‖𝐳ℓ‖22​𝟏\+b0ℓ\)\\theta^\{\\ell\}\\leftarrow\\sigma\\\!\\left\(W\_\{0\}^\{\\ell\}\\mathbf\{z\}\_\{\\ell\}\+\\alpha\\left\\lVert\\mathbf\{z\}\_\{\\ell\}\\right\\rVert\_\{2\}^\{2\}\\mathbf\{1\}\+b\_\{0\}^\{\\ell\}\\right\)
7:endfor

8:

θ←\(\(θ1,…,θL\),θnorm\)\\theta\\leftarrow\\bigl\(\(\\theta^\{1\},\\dots,\\theta^\{L\}\),\\,\\theta\_\{\\mathrm\{norm\}\}\\bigr\)
9:return

fθf\_\{\\theta\}⊳\\trianglerightinference is an ordinary dense forward pass

#### Remark \(regimes\)\.

L=1L=1recovers single latent vector training, where one latent generates every mapped parameter, andL\>1L\>1recovers layer\-wise training with one latent per layer\. Only the loop bounds change\.

#### Remark \(fine\-tuning\)\.

To adapt a pretrained network, replace the seed\-generated centerb0ℓb\_\{0\}^\{\\ell\}by the corresponding block of the pretrained parametersθpre\\theta\_\{\\mathrm\{pre\}\}and takeσ\\sigmato be the identity, so that the latent encodes a displacementθft=θpre\+W0​𝐳\\theta\_\{\\mathrm\{ft\}\}=\\theta\_\{\\mathrm\{pre\}\}\+W\_\{0\}\\mathbf\{z\}\. The artifact then carries a short identifierhhof the base model,𝒜ft=\(h,s,\{𝐳^ℓ\},θnorm\)\\mathcal\{A\}\_\{\\mathrm\{ft\}\}=\(h,s,\\\{\\hat\{\\mathbf\{z\}\}\_\{\\ell\}\\\},\\theta\_\{\\mathrm\{norm\}\}\), and the base is referenced rather than stored\.

## Appendix BAppendix: Hyperparameter values for the experiments for reproducibility

#### CNN\.

Latents are trained for3030epochs with Adam at learning rate10−210^\{\-2\}and batch size128128, under a cosine schedule with100100warmup steps decaying to5%5\\%of the peak rate, followed by22epochs of quantization\-aware fine\-tuning with Adam at2×10−32\\times 10^\{\-3\}under a cosine schedule without warmup; the full\-network baselines use Adam at10−310^\{\-3\}, batch size256256, and2020epochs with200200warmup steps\. All forward and backward passes run underbfloat16autocast, and no gradient clipping is used\.

#### LSTM\.

Latents are trained for6060epochs with Adam at learning rate10−210^\{\-2\}and batch size256256, under a cosine schedule with100100warmup steps decaying to5%5\\%of the peak rate, followed by33epochs of quantization\-aware fine\-tuning with Adam at2×10−32\\times 10^\{\-3\}at a constant rate; the full\-network baseline uses Adam at10−310^\{\-3\}, batch size256256, and4040epochs with100100warmup steps\. All forward and backward passes run underbfloat16autocast, and no gradient clipping is used\.

#### MLP\.

Latents are trained for3030epochs with Adam at learning rate10−210^\{\-2\}and batch size512512, under a cosine schedule with100100warmup steps decaying to5%5\\%of the peak rate, followed by22epochs of quantization\-aware fine\-tuning with Adam at2×10−32\\times 10^\{\-3\}under a cosine schedule without warmup; the full\-network baselines use AdamW at10−310^\{\-3\}, batch size512512, and2020epochs with200200warmup steps\. All forward and backward passes run underbfloat16autocast, and no gradient clipping is used\.

#### Fine\-tuning\.

Latents are trained for3030epochs with Adam at learning rate5×10−25\\times 10^\{\-2\}and batch size3232, under a cosine schedule with100100warmup steps decaying to5%5\\%of the peak rate, with gradients clipped to norm1\.01\.0, which is required for stability at the largest latent budget, followed by22epochs of quantization\-aware fine\-tuning with Adam at5×10−35\\times 10^\{\-3\}at a constant rate and the same clipping; the full fine\-tuning baseline uses Adam at3×10−43\\times 10^\{\-4\}, batch size3232, and1515epochs with100100warmup steps\. All forward and backward passes run underbfloat16autocast\.

## Appendix CAppendix: Experiment results with 3 seeds variability

The tables in this Section repeat the results of Section 5 with a standard deviation attached to every reported quantity\. The reported value in each case is the same mean given in the main text, with the standard deviation reported across 3 random seeds\. A random seed redraws the trained latent, the batch order, and the seeded basis and center\. Artifact sizes are deterministic and carry no variability, so they are not repeated here\.

Table 11:Near\-orthonormality of the seeded basis: Gram statistics ofW0⊤​W0W\_\{0\}^\{\\top\}W\_\{0\}over a sample of256256columns, for the Gaussian and Rademacher constructions at three parameter countsPP\.Table 12:CNN Results: Test accuracy \(mean%±\\pmstandard deviation%\) for the two targets trained conventionally \(“full”\) and as mapped Kilobyte models under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT, on MNIST and FashionMNIST\.Table 13:LSTM Results on the Beijing air\-quality dataset: test MSE \(×10−3\\times 10^\{\-3\}\) for the LSTM trained conventionally \(“full”\) and as a mapped Kilobyte model under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT\.Table 14:MLP Results on two tabular datasets: test accuracy \(mean%±\\pmstandard deviation%\) for MLP1 and MLP2 trained conventionally \(“full”\) and as mapped Kilobyte models under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT\.Table 15:Fine\-tuning Results: Pets test accuracy \(%\) for an ImageNet\-pretrained ResNet\-50 adapted by a mapped delta under SLVT and LWT, at latent budgetddand int8/int4 under PTQ and QAT\. The baseline is full fine\-tuning\.