p-Spin Glass Network Efficient Single-Batch Continual Learning

# p-Spin Glass Network Efficient Single-Batch Continual Learning
Source: [https://arxiv.org/html/2608.14774](https://arxiv.org/html/2608.14774)
###### Abstract

Modern sequence models heavily rely on massive memory footprints and large\-batch stochastic optimization, barriers that restrict sample efficiency and continual learning\. We introduce thepp\-Spin Glass Network, a novel architecture that overcomes these limitations, structurally manages optimization variance and yields four noticeable capabilities: 1\. It enforces memory efficiency: native ternary quantization compresses internal parameters by8×8\\times, while exact implicit gradients strictly bound activation memory to𝒪⁡\(B⋅T⋅D\)\\mathcal\{O\}\(B\\cdot T\\cdot D\)\. 2\. it demonstrates sample efficiency, matching the asymptotic performance of a Transformer baseline while utilizing8×8\\timesfewer training sequences\. 3\. Method enables single\-batch stability and smooth, monotonic convergence at a stochastic micro\-batch size of11\. 4\. Finally, this stability proves modality\-agnostic, maintaining robust temporal credit assignment across both discrete subword and long horizon uncompressed raw byte streams\. Ultimately, this work removes large batch requirement for stable deep learning, establishing a foundation for continuous learning and edge AI\.Code:[https://github\.com/VladimerKhasia/sgn](https://github.com/VladimerKhasia/sgn)

###### Keywords:

Machine Learning, Sequence Models, Spin Glass

††affiliationnotice:Preprint\.![Refer to caption](https://arxiv.org/html/2608.14774v1/visualisation.png)Figure 1:Architectural diagram of thepp\-Spin Glass Network\. The left column illustrates modality\-agnostic input encoding\. The center column details the hardware\-fused continuous state\-space routing and the higher\-orderpp\-spin attractor search\. The right column depicts the final vocabulary projection and cross\-entropy evaluation\.## 1Introduction

Modern deep learning relies fundamentally on the Transformer architecture\([18](https://arxiv.org/html/2608.14774#bib.bib5);[16](https://arxiv.org/html/2608.14774#bib.bib6)\), whose dependence on massive memory footprints and large batch gradient accumulation heavily obstructs low resource and continual learning\([14](https://arxiv.org/html/2608.14774#bib.bib7)\)\. While recent advancements in linear\-time state\-space models \(SSMs\)\([8](https://arxiv.org/html/2608.14774#bib.bib8);[4](https://arxiv.org/html/2608.14774#bib.bib9);[7](https://arxiv.org/html/2608.14774#bib.bib10)\)alleviate sequence length memory bottlenecks and extreme network quantization\([19](https://arxiv.org/html/2608.14774#bib.bib11);[12](https://arxiv.org/html/2608.14774#bib.bib12)\)compresses parameter footprints, these techniques collectively fail to resolve the severe gradient variance and optimization instability inherent to training at minimal batch sizes\.

To overcome this stochastic optimization barrier, we reframe sequence representation through the theoretical lens of implicit deep learning\([1](https://arxiv.org/html/2608.14774#bib.bib13);[5](https://arxiv.org/html/2608.14774#bib.bib14)\)and physics\-inspired thermodynamic attractors\([9](https://arxiv.org/html/2608.14774#bib.bib15);[2](https://arxiv.org/html/2608.14774#bib.bib16);[17](https://arxiv.org/html/2608.14774#bib.bib17)\)\. Unlike standard architectures that map inputs through explicitly unrolled feed\-forward graphs, implicit models define hidden states as the converged roots of an iterative fixed point solver\. By extracting exact analytical gradients at the equilibrium manifold via the Implicit Function Theorem \(IFT\), activation memory becomes mathematically decoupled from the integration depth\.

Building upon these principles while inspired by physics of Spin Glasses\([13](https://arxiv.org/html/2608.14774#bib.bib1);[15](https://arxiv.org/html/2608.14774#bib.bib2);[3](https://arxiv.org/html/2608.14774#bib.bib3);[6](https://arxiv.org/html/2608.14774#bib.bib4)\), we introduce thepp\-Spin Glass Network\. We apply ternary quantization specifically to the internal thermodynamic projection matrices\. This targeted compression explicitly bounds the Lipschitz constant of the equilibrium layer, providing a structural thermodynamic regularization that dampens gradient variance\. Formulated as a generalized framework, the architecture sustains robust temporal credit assignment across both standard subword token spaces and long horizon uncompressed raw byte streams\([20](https://arxiv.org/html/2608.14774#bib.bib18);[21](https://arxiv.org/html/2608.14774#bib.bib19);[10](https://arxiv.org/html/2608.14774#bib.bib20)\)\.

In summary, core contributions are as follows:

- •Architectural Memory Efficiency:We quantize the projection matrices of the implicitpp\-spin attractor \(Wext,WintW\_\{\\text\{ext\}\},W\_\{\\text\{int\}\}\) to ternary values, yielding an8×8\\timesparameter compression in the equilibrium layers\. Furthermore, by coupling SRAM\-fused state\-space chunking with gradient derivation via the Implicit Function Theorem, we bound HBM activation memory strictly to𝒪⁡\(B⋅T⋅D\)\\mathcal\{O\}\(B\\cdot T\\cdot D\), unconditionally invariant to the fixed point iteration countKK\.
- •Sample\-Efficient Optimization:We demonstrate that the method achieves asymptotic performance parity with optimized standard Transformers while consuming only12\.5%12\.5\\%of the required training sequences\.
- •Single\-Batch Stability for Continual Learning:We establish that the bounded fixed point solver structurally dampens gradient variance\. This mathematical regularization guarantees smooth, monotonic convergence even at a pure stochastic micro batch size ofB=1B=1, eliminating the standard requirement for massive batch accumulation\.
- •Modality Agnostic Processing:We empirically validate the architecture as a generalized sequence modeling framework\. It successfully maintains temporal credit assignment across fundamentally different representation scales, from discrete subword token spaces \(\|V\|=49152\|V\|=49152\) to extreme\-horizon continuous raw byte streams \(\|V\|=256\|V\|=256\)\.

## 2Methodology

### 2\.1Mathematical Formulation and Derivation

This section provides formulation of thepp\-Spin Glass Network\. The architecture acts as a general sequence modeling framework, evaluated here on discrete language modalities\. The forward and backward propagation dynamics strictly couple continuous time state\-space integrations withpp\-spin glass equilibrium attractors\.

Notation Glossary:

- •B,T,DB,T,D: Batch size, sequence length, and hidden feature dimension\.
- •H,dhH,d\_\{h\}: Number of attention heads and head dimension \(D=H⋅dhD=H\\cdot d\_\{h\}\)\.
- •C,NC,N: Chunk size and number of sequence chunks \(T=N⋅CT=N\\cdot C\)\.
- •X∈ℝB×T×DX\\in\\mathbb\{R\}^\{B\\times T\\times D\}: Input sequence representation\.
- •Wq,Wk,Wv,Wg,Wo∈ℝD×DW\_\{q\},W\_\{k\},W\_\{v\},W\_\{g\},W\_\{o\}\\in\\mathbb\{R\}^\{D\\times D\}: State\-space linear projection weights\.
- •Wext∈ℝ2​Dint×DW\_\{\\text\{ext\}\}\\in\\mathbb\{R\}^\{2D\_\{\\text\{int\}\}\\times D\},Wint∈ℝ2​Dint×DintW\_\{\\text\{int\}\}\\in\\mathbb\{R\}^\{2D\_\{\\text\{int\}\}\\times D\_\{\\text\{int\}\}\}: External drive and internal equilibrium ternary matrices\.
- •γparam∈ℝDint\\gamma\_\{\\text\{param\}\}\\in\\mathbb\{R\}^\{D\_\{\\text\{int\}\}\}: Learnable spin frustration \(relaxation rate\) parameters\.
- •K∈ℤ\+K\\in\\mathbb\{Z\}\_\{\+\}: Number of equilibrium micro\-steps \(fixed point iterations\)\.

#### 2\.1\.1Continuous ODE Integration via Gated State\-Spaces

For a given layerll, the input tensorX\(l−1\)X^\{\(l\-1\)\}is normalized via Root Mean Square Normalization \(withϵ=10−6\\epsilon=10^\{\-6\}\):

X~b,t,d=Xb,t,d\(l−1\)1D​∑j=1D\(Xb,t,j\(l−1\)\)2\+ϵ⊙Wnorm,d\\tilde\{X\}\_\{b,t,d\}=\\frac\{X\_\{b,t,d\}^\{\(l\-1\)\}\}\{\\sqrt\{\\frac\{1\}\{D\}\\sum\_\{j=1\}^\{D\}\(X\_\{b,t,j\}^\{\(l\-1\)\}\)^\{2\}\+\\epsilon\}\}\\odot W\_\{\\text\{norm\},d\}\(1\)
The state\-space projectionsQ,K,V∈ℝB×T×H×dhQ,K,V\\in\\mathbb\{R\}^\{B\\times T\\times H\\times d\_\{h\}\}are computed asX~​Wq⊤,X~​Wk⊤,X~​Wv⊤\\tilde\{X\}W\_\{q\}^\{\\top\},\\tilde\{X\}W\_\{k\}^\{\\top\},\\tilde\{X\}W\_\{v\}^\{\\top\}, withQ←Q⋅dh−0\.5Q\\leftarrow Q\\cdot d\_\{h\}^\{\-0\.5\}\. The data\-dependent ODE decay gateG∈ℝB×T×HG\\in\\mathbb\{R\}^\{B\\times T\\times H\}bounds the continuous\-time temporal growth strictly to negative domains via the softplus operator:

Gb,t,h=1dh∑j=1dh−log\(1\+exp\(\(X~Wg⊤\)b,t,h,j\)\)G\_\{b,t,h\}=\\frac\{1\}\{d\_\{h\}\}\\sum\_\{j=1\}^\{d\_\{h\}\}\-\\log\\left\(1\+\\exp\\left\(\(\\tilde\{X\}W\_\{g\}^\{\\top\}\)\_\{b,t,h,j\}\\right\)\\right\)\(2\)
To circumventO⁡\(T2\)O\(T^\{2\}\)memory materialization, the sequence is partitioned intoNNchunks of lengthCC\. The intra\-chunk cumulative sumGc​sG\_\{cs\}governs the exact causal decay maskℳ\(i\)∈ℝC×C\\mathcal\{M\}^\{\(i\)\}\\in\\mathbb\{R\}^\{C\\times C\}, materialized directly in SRAM:

ℳc,j\(i\)=\{exp⁡\(Gc​s\(i,c\)−Gc​s\(i,j\)\)if​c≥j0otherwise\\mathcal\{M\}^\{\(i\)\}\_\{c,j\}=\\begin\{cases\}\\exp\\left\(G\_\{cs\}^\{\(i,c\)\}\-G\_\{cs\}^\{\(i,j\)\}\\right\)&\\text\{if \}c\\geq j\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(3\)The contextualized output of chunkiicombines intra chunk causal attention and cross chunk global memoryh\(i−1\)h^\{\(i\-1\)\}:

Ointra\(i\)\\displaystyle O\_\{\\text\{intra\}\}^\{\(i\)\}=\(Q\(i\)​\(K\(i\)\)⊤⊙ℳ\(i\)\)​V\(i\)\\displaystyle=\\left\(Q^\{\(i\)\}\(K^\{\(i\)\}\)^\{\\top\}\\odot\\mathcal\{M\}^\{\(i\)\}\\right\)V^\{\(i\)\}\(4\)Ocross\(i\)\\displaystyle O\_\{\\text\{cross\}\}^\{\(i\)\}=\(Q\(i\)​h\(i−1\)\)⊙exp⁡\(Gc​s\(i\)\)\\displaystyle=\\left\(Q^\{\(i\)\}h^\{\(i\-1\)\}\\right\)\\odot\\exp\\left\(G\_\{cs\}^\{\(i\)\}\\right\)\(5\)h\(i\)\\displaystyle h^\{\(i\)\}=h\(i−1\)⊙exp⁡\(Gc​s\(i,C\)\)\\displaystyle=h^\{\(i\-1\)\}\\odot\\exp\\left\(G\_\{cs\}^\{\(i,C\)\}\\right\)\+\(K\(i\)⊙exp⁡\(Gc​s\(i,C\)−Gc​s\(i\)\)\)⊤​V\(i\)\\displaystyle\\quad\+\\left\(K^\{\(i\)\}\\odot\\exp\\left\(G\_\{cs\}^\{\(i,C\)\}\-G\_\{cs\}^\{\(i\)\}\\right\)\\right\)^\{\\top\}V^\{\(i\)\}\(6\)The concatenated sequence passes throughWo⊤W\_\{o\}^\{\\top\}, yieldingXssmX\_\{\\text\{ssm\}\}\. The residual update isXmid=X\(l−1\)\+XssmX\_\{\\text\{mid\}\}=X^\{\(l\-1\)\}\+X\_\{\\text\{ssm\}\}\.

#### 2\.1\.2Higher\-Order Interactions \(pp\-Spin Glass Gated Attractors\)

The residual tensorXmidX\_\{\\text\{mid\}\}is normalized toX¯\\bar\{X\}\. Thepp\-spin glass MLP evaluates an implicit equilibrium fixed point\. To enforce parameter efficiency, the dense projection matrices are quantized to\{−1,0,1\}\\\{\-1,0,1\\\}using an absolute mean scalessand a Straight\-Through Estimator \(STE\)\. ForW∈\{Wext,Wint\}W\\in\\\{W\_\{\\text\{ext\}\},W\_\{\\text\{int\}\}\\\}:

s\\displaystyle s=max⁡\(1\|W\|​∑w∈W\|w\|,10−5\)\\displaystyle=\\max\\left\(\\frac\{1\}\{\|W\|\}\\sum\_\{w\\in W\}\|w\|,10^\{\-5\}\\right\)\(7\)WQ\\displaystyle W^\{Q\}=STE​\(clamp​\(round​\(Ws\),−1,1\)⋅s\)\\displaystyle=\\text\{STE\}\\left\(\\text\{clamp\}\\left\(\\text\{round\}\\left\(\\frac\{W\}\{s\}\\right\),\-1,1\\right\)\\cdot s\\right\)\(8\)
The external thermodynamic drive projects the representation to a2​Dint2D\_\{\\text\{int\}\}space:Z=X¯​\(WextQ\)⊤Z=\\bar\{X\}\(W\_\{\\text\{ext\}\}^\{Q\}\)^\{\\top\}\. SplittingZZisolates the value boundsU0U\_\{0\}and the spin gatesG0G\_\{0\}\. The frustration rate is bounded byγ=0\.5​tanh⁡\(γparam\)\\gamma=0\.5\\tanh\(\\gamma\_\{\\text\{param\}\}\)\. Fork∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}micro steps, the network seeks the energy minimumY∗=YKY^\{\*\}=Y\_\{K\}:

Yk−1\\displaystyle Y\_\{k\-1\}=tanh⁡\(Uk−1\)⊙σ⁡\(Gk−1\)\\displaystyle=\\tanh\(U\_\{k\-1\}\)\\odot\\sigma\(G\_\{k\-1\}\)\(9\)Hk\\displaystyle H\_\{k\}=Z\+\(γ⊙Yk−1\)​\(WintQ\)⊤\\displaystyle=Z\+\(\\gamma\\odot Y\_\{k\-1\}\)\(W\_\{\\text\{int\}\}^\{Q\}\)^\{\\top\}\(10\)Uk,Gk\\displaystyle U\_\{k\},G\_\{k\}=Hk,\[:,:,:Dint\],Hk,\[:,:,Dint:\]\\displaystyle=H\_\{k,\[:,:,:D\_\{\\text\{int\}\}\]\},H\_\{k,\[:,:,D\_\{\\text\{int\}\}:\]\}\(11\)The equilibrium stateY∗Y^\{\*\}is down\-projected viaWdownW\_\{\\text\{down\}\}, producing the layer’s final output:X\(l\)=Xmid\+Y∗​Wdown⊤X^\{\(l\)\}=X\_\{\\text\{mid\}\}\+Y^\{\*\}W\_\{\\text\{down\}\}^\{\\top\}\.

#### 2\.1\.3Exact Implicit Gradients

Backpropagation through the iterative attractor relies on the Implicit Function Theorem via a Neumann series approximation, ensuring exact gradient reconstruction withoutO⁡\(K\)O\(K\)graph unrolling\. Defining equilibrium derivativesϕu=\(1−tanh2⁡\(UK\)\)⊙σ⁡\(GK\)\\phi\_\{u\}=\(1\-\\tanh^\{2\}\(U\_\{K\}\)\)\\odot\\sigma\(G\_\{K\}\)andϕg=tanh⁡\(UK\)⊙σ⁡\(GK\)⊙\(1−σ⁡\(GK\)\)\\phi\_\{g\}=\\tanh\(U\_\{K\}\)\\odot\\sigma\(G\_\{K\}\)\\odot\(1\-\\sigma\(G\_\{K\}\)\), the iterative gradientVkV\_\{k\}given incoming∇Y∗ℒ\\nabla\_\{Y^\{\*\}\}\\mathcal\{L\}is:

Vk=∇Y∗ℒ\+γ⊙\(\[Vk−1⊙ϕu∥Vk−1⊙ϕg\]WintQ\)V\_\{k\}=\\nabla\_\{Y^\{\*\}\}\\mathcal\{L\}\+\\gamma\\odot\\left\(\\left\[V\_\{k\-1\}\\odot\\phi\_\{u\}\\parallel V\_\{k\-1\}\\odot\\phi\_\{g\}\\right\]W\_\{\\text\{int\}\}^\{Q\}\\right\)\(12\)Yielding exact gradients∇H=\[\(VK⊙ϕu\)∥\(VK⊙ϕg\)\]\\nabla\_\{H\}=\[\\left\(V\_\{K\}\\odot\\phi\_\{u\}\\right\)\\parallel\\left\(V\_\{K\}\\odot\\phi\_\{g\}\\right\)\], strictly evaluated at the fixed point manifold\.

### 2\.2Algorithmic Specification

We present the complete algorithmic specifications of thepp\-Spin Glass Network\. To demonstrate its modality agnostic applicability as a general sequence modeling framework, the methodology is provided in two distinct, formulations corresponding directly to the provided code implementations\.

Algorithm[1](https://arxiv.org/html/2608.14774#alg1)delineates the standard subword\-tokenized architecture leveraging a unified vocabulary size of\|V\|=49152\|V\|=49152\. Conversely, Algorithm[2](https://arxiv.org/html/2608.14774#alg2)formalizes the character level \(continuous raw byte\) formulation operating over an uncompressed vocabulary of\|V\|=256\|V\|=256\. Both algorithms execute the exact same internalpp\-spin attractor and hardware\-fused state\-space dynamics, diverging exclusively in their input encoding spaces, embedding matrices, and final evaluation metrics \(Perplexity versus Bits Per Byte\)\.

Algorithm 1pp\-Spin Glass Network \(Subword Level\)0:Raw Text

SS, Subword Tokenizer

τsub\\tau\_\{\\text\{sub\}\}\(

\|V\|=49152\|V\|=49152\), Target

YtargY\_\{\\text\{targ\}\}, Layers

LL\.

0:Loss

ℒ\\mathcal\{L\}, Perplexity metric\.

1:

I←τsub​\(S\)∈ℤB×TI\\leftarrow\\tau\_\{\\text\{sub\}\}\(S\)\\in\\mathbb\{Z\}^\{B\\times T\}
2:

X\(0\)←Embedding​\(I,Wemb\)X^\{\(0\)\}\\leftarrow\\text\{Embedding\}\(I,W\_\{\\text\{emb\}\}\)
3:for

l=1l=1to

LLdo

4:// 1\. Gated State\-Space Continuous Integration

5:

X~←RMSNorm​\(X\(l−1\)\)\\tilde\{X\}\\leftarrow\\text\{RMSNorm\}\(X^\{\(l\-1\)\}\)
6:

Q,K,V←X~​Wq⊤,X~​Wk⊤,X~​Wv⊤Q,K,V\\leftarrow\\tilde\{X\}W\_\{q\}^\{\\top\},\\tilde\{X\}W\_\{k\}^\{\\top\},\\tilde\{X\}W\_\{v\}^\{\\top\}
7:

G←1dh∑−log\(1\+exp\(X~Wg⊤\)\)G\\leftarrow\\frac\{1\}\{d\_\{h\}\}\\sum\-\\log\(1\+\\exp\(\\tilde\{X\}W\_\{g\}^\{\\top\}\)\)
8:Split into chunks of size

CC;

h←𝟎h\\leftarrow\\mathbf\{0\}
9:foreach chunk

i∈\{1​…​N\}i\\in\\\{1\\dots N\\\}do

10:

ℳ\(i\)←exp⁡\(Gc​s\(i\)−\(Gc​s\(i\)\)⊤\)\\mathcal\{M\}^\{\(i\)\}\\leftarrow\\exp\(G\_\{cs\}^\{\(i\)\}\-\(G\_\{cs\}^\{\(i\)\}\)^\{\\top\}\)\(Masked lower\-triangular\)

11:

Ointra←\(\(Qi​Ki⊤\)⊙ℳ\(i\)\)​ViO\_\{\\text\{intra\}\}\\leftarrow\(\(Q\_\{i\}K\_\{i\}^\{\\top\}\)\\odot\\mathcal\{M\}^\{\(i\)\}\)V\_\{i\}
12:

Ocross←\(Qi​h\)⊙exp⁡\(Gc​s\(i\)\)O\_\{\\text\{cross\}\}\\leftarrow\(Q\_\{i\}h\)\\odot\\exp\(G\_\{cs\}^\{\(i\)\}\)
13:

Oi←Ointra\+OcrossO\_\{i\}\\leftarrow O\_\{\\text\{intra\}\}\+O\_\{\\text\{cross\}\}
14:Update continuous state

hhusing

Ki,Vi,Gc​sK\_\{i\},V\_\{i\},G\_\{cs\}
15:endfor

16:

Xmid←X\(l−1\)\+Concat​\(O1​…​ON\)​Wo⊤X\_\{\\text\{mid\}\}\\leftarrow X^\{\(l\-1\)\}\+\\text\{Concat\}\(O\_\{1\}\\dots O\_\{N\}\)W\_\{o\}^\{\\top\}
17:// 2\. Higher\-Order Interactions \(pp\-Spin Equilibrium\)

18:

X¯←RMSNorm​\(Xmid\)\\bar\{X\}\\leftarrow\\text\{RMSNorm\}\(X\_\{\\text\{mid\}\}\)
19:

WextQ,WintQ←Quantize1\.58b​\(Wext\),Quantize1\.58b​\(Wint\)W\_\{\\text\{ext\}\}^\{Q\},W\_\{\\text\{int\}\}^\{Q\}\\leftarrow\\text\{Quantize1\.58b\}\(W\_\{\\text\{ext\}\}\),\\text\{Quantize1\.58b\}\(W\_\{\\text\{int\}\}\)
20:

Z←X¯​\(WextQ\)⊤Z\\leftarrow\\bar\{X\}\(W\_\{\\text\{ext\}\}^\{Q\}\)^\{\\top\}
21:

U0,G0←Split​\(Z,dim=−1\)U\_\{0\},G\_\{0\}\\leftarrow\\text\{Split\}\(Z,\\text\{dim\}=\-1\)
22:

Y←tanh⁡\(U0\)⊙σ⁡\(G0\)Y\\leftarrow\\tanh\(U\_\{0\}\)\\odot\\sigma\(G\_\{0\}\)
23:for

k=1k=1to

KKdo

24:

Hk←Z\+\(γ⊙Y\)​\(WintQ\)⊤H\_\{k\}\\leftarrow Z\+\(\\gamma\\odot Y\)\(W\_\{\\text\{int\}\}^\{Q\}\)^\{\\top\}
25:

Uk,Gk←Split​\(Hk,dim=−1\)U\_\{k\},G\_\{k\}\\leftarrow\\text\{Split\}\(H\_\{k\},\\text\{dim\}=\-1\)
26:

Y←tanh⁡\(Uk\)⊙σ⁡\(Gk\)Y\\leftarrow\\tanh\(U\_\{k\}\)\\odot\\sigma\(G\_\{k\}\)
27:endfor

28:

X\(l\)←Xmid\+Y​Wdown⊤X^\{\(l\)\}\\leftarrow X\_\{\\text\{mid\}\}\+YW\_\{\\text\{down\}\}^\{\\top\}
29:endfor

30:// 3\. Vocabulary Projection \(Tied Embeddings\) & Loss

31:

Y^←RMSNorm​\(X\(L\)\)​Wemb⊤\\hat\{Y\}\\leftarrow\\text\{RMSNorm\}\(X^\{\(L\)\}\)W\_\{\\text\{emb\}\}^\{\\top\}\{

WembW\_\{\\text\{emb\}\}tied to output head\}

32:

ℒ←CrossEntropy​\(Y^,Ytarg\)\\mathcal\{L\}\\leftarrow\\text\{CrossEntropy\}\(\\hat\{Y\},Y\_\{\\text\{targ\}\}\)
33:return

ℒ\\mathcal\{L\},Perplexity

←exp⁡\(ℒ\)\\leftarrow\\exp\(\\mathcal\{L\}\)

Algorithm 2pp\-Spin Glass Network \(Byte Level\)0:Raw Text

SS, UTF\-8 Byte Encoder

τbyte\\tau\_\{\\text\{byte\}\}\(

\|V\|=256\|V\|=256\), Target

YtargY\_\{\\text\{targ\}\}, Layers

LL\.

0:Loss

ℒ\\mathcal\{L\}, Bits\-Per\-Byte \(BPB\) metric\.

1:

I←τbyte​\(S\)∈\{0​…​255\}B×TI\\leftarrow\\tau\_\{\\text\{byte\}\}\(S\)\\in\\\{0\\dots 255\\\}^\{B\\times T\}
2:

X\(0\)←Embedding​\(I,Wemb\)X^\{\(0\)\}\\leftarrow\\text\{Embedding\}\(I,W\_\{\\text\{emb\}\}\)
3:for

l=1l=1to

LLdo

4:// 1\. Gated State\-Space Continuous Integration

5:

X~←RMSNorm​\(X\(l−1\)\)\\tilde\{X\}\\leftarrow\\text\{RMSNorm\}\(X^\{\(l\-1\)\}\)
6:

Q,K,V←X~​Wq⊤,X~​Wk⊤,X~​Wv⊤Q,K,V\\leftarrow\\tilde\{X\}W\_\{q\}^\{\\top\},\\tilde\{X\}W\_\{k\}^\{\\top\},\\tilde\{X\}W\_\{v\}^\{\\top\}
7:

G←1dh∑−log\(1\+exp\(X~Wg⊤\)\)G\\leftarrow\\frac\{1\}\{d\_\{h\}\}\\sum\-\\log\(1\+\\exp\(\\tilde\{X\}W\_\{g\}^\{\\top\}\)\)
8:Split into chunks of size

CC;

h←𝟎h\\leftarrow\\mathbf\{0\}
9:foreach chunk

i∈\{1​…​N\}i\\in\\\{1\\dots N\\\}do

10:

ℳ\(i\)←exp⁡\(Gc​s\(i\)−\(Gc​s\(i\)\)⊤\)\\mathcal\{M\}^\{\(i\)\}\\leftarrow\\exp\(G\_\{cs\}^\{\(i\)\}\-\(G\_\{cs\}^\{\(i\)\}\)^\{\\top\}\)\(Masked lower\-triangular\)

11:

Ointra←\(\(Qi​Ki⊤\)⊙ℳ\(i\)\)​ViO\_\{\\text\{intra\}\}\\leftarrow\(\(Q\_\{i\}K\_\{i\}^\{\\top\}\)\\odot\\mathcal\{M\}^\{\(i\)\}\)V\_\{i\}
12:

Ocross←\(Qi​h\)⊙exp⁡\(Gc​s\(i\)\)O\_\{\\text\{cross\}\}\\leftarrow\(Q\_\{i\}h\)\\odot\\exp\(G\_\{cs\}^\{\(i\)\}\)
13:

Oi←Ointra\+OcrossO\_\{i\}\\leftarrow O\_\{\\text\{intra\}\}\+O\_\{\\text\{cross\}\}
14:Update continuous state

hhusing

Ki,Vi,Gc​sK\_\{i\},V\_\{i\},G\_\{cs\}
15:endfor

16:

Xmid←X\(l−1\)\+Concat​\(O1​…​ON\)​Wo⊤X\_\{\\text\{mid\}\}\\leftarrow X^\{\(l\-1\)\}\+\\text\{Concat\}\(O\_\{1\}\\dots O\_\{N\}\)W\_\{o\}^\{\\top\}
17:// 2\. Higher\-Order Interactions \(pp\-Spin Equilibrium\)

18:

X¯←RMSNorm​\(Xmid\)\\bar\{X\}\\leftarrow\\text\{RMSNorm\}\(X\_\{\\text\{mid\}\}\)
19:

WextQ,WintQ←Quantize1\.58b​\(Wext\),Quantize1\.58b​\(Wint\)W\_\{\\text\{ext\}\}^\{Q\},W\_\{\\text\{int\}\}^\{Q\}\\leftarrow\\text\{Quantize1\.58b\}\(W\_\{\\text\{ext\}\}\),\\text\{Quantize1\.58b\}\(W\_\{\\text\{int\}\}\)
20:

Z←X¯​\(WextQ\)⊤Z\\leftarrow\\bar\{X\}\(W\_\{\\text\{ext\}\}^\{Q\}\)^\{\\top\}
21:

U0,G0←Split​\(Z,dim=−1\)U\_\{0\},G\_\{0\}\\leftarrow\\text\{Split\}\(Z,\\text\{dim\}=\-1\)
22:

Y←tanh⁡\(U0\)⊙σ⁡\(G0\)Y\\leftarrow\\tanh\(U\_\{0\}\)\\odot\\sigma\(G\_\{0\}\)
23:for

k=1k=1to

KKdo

24:

Hk←Z\+\(γ⊙Y\)​\(WintQ\)⊤H\_\{k\}\\leftarrow Z\+\(\\gamma\\odot Y\)\(W\_\{\\text\{int\}\}^\{Q\}\)^\{\\top\}
25:

Uk,Gk←Split​\(Hk,dim=−1\)U\_\{k\},G\_\{k\}\\leftarrow\\text\{Split\}\(H\_\{k\},\\text\{dim\}=\-1\)
26:

Y←tanh⁡\(Uk\)⊙σ⁡\(Gk\)Y\\leftarrow\\tanh\(U\_\{k\}\)\\odot\\sigma\(G\_\{k\}\)
27:endfor

28:

X\(l\)←Xmid\+Y​Wdown⊤X^\{\(l\)\}\\leftarrow X\_\{\\text\{mid\}\}\+YW\_\{\\text\{down\}\}^\{\\top\}
29:endfor

30:// 3\. Vocabulary Projection \(Tied Embeddings\) & Loss

31:

Y^←RMSNorm​\(X\(L\)\)​Wemb⊤\\hat\{Y\}\\leftarrow\\text\{RMSNorm\}\(X^\{\(L\)\}\)W\_\{\\text\{emb\}\}^\{\\top\}\{

WembW\_\{\\text\{emb\}\}tied to output head\}

32:

ℒ←CrossEntropy​\(Y^,Ytarg\)\\mathcal\{L\}\\leftarrow\\text\{CrossEntropy\}\(\\hat\{Y\},Y\_\{\\text\{targ\}\}\)
33:return

ℒ\\mathcal\{L\},BPB

←ℒ/ln⁡\(2\)\\leftarrow\\mathcal\{L\}/\\ln\(2\)

### 2\.3Complete Complexity Analysis

We analyze the theoretical time \(computational operations\) and space \(memory footprint\) complexities of thepp\-Spin Glass Network\. The analysis strictly isolates the discrete modalities \(Subword vs\. Byte level\) to formalize how vocabulary scaling impacts the algorithmic execution bounds\. LetLLdenote the number of layers\.

#### 2\.3\.1Time Complexity \(Computational Operations\)

The forward pass time complexity is governed by three primary computational phases: the continuous state\-space integration, thepp\-spin equilibrium fixed point derivation, and the final vocabulary projection\.

1\. Gated State\-Space Continuous Integration:The initial linear projections \(Q,K,V,gQ,K,V,g\) require𝒪⁡\(B⋅T⋅D2\)\\mathcal\{O\}\(B\\cdot T\\cdot D^\{2\}\)operations\. The sequence is processed inN=T/CN=T/Cchunks\. The intra\-chunk masked attention \(evaluated in SRAM\) computes aC×CC\\times Ctemporal interaction matrix per head, requiring𝒪⁡\(C2⋅dh\)\\mathcal\{O\}\(C^\{2\}\\cdot d\_\{h\}\)operations per chunk\. AcrossNNchunks andHHheads, this yields𝒪⁡\(N⋅H⋅C2⋅dh\)=𝒪⁡\(B⋅T⋅C⋅D\)\\mathcal\{O\}\(N\\cdot H\\cdot C^\{2\}\\cdot d\_\{h\}\)=\\mathcal\{O\}\(B\\cdot T\\cdot C\\cdot D\)\. The cross\-chunk continuous state update involves multiplication with thedh×dhd\_\{h\}\\times d\_\{h\}hidden statehh, taking𝒪⁡\(C⋅dh2\)\\mathcal\{O\}\(C\\cdot d\_\{h\}^\{2\}\)per chunk\. Across all heads and chunks, this requires𝒪⁡\(B⋅T⋅D⋅dh\)\\mathcal\{O\}\(B\\cdot T\\cdot D\\cdot d\_\{h\}\)\. The total time complexity for the state\-space module per layer is:

𝒪ssm=𝒪⁡\(B⋅T⋅D⋅\(D\+C\+dh\)\)\\mathcal\{O\}\_\{\\text\{ssm\}\}=\\mathcal\{O\}\\big\(B\\cdot T\\cdot D\\cdot\(D\+C\+d\_\{h\}\)\\big\)\(13\)
2\. Higher\-Order Interactions \(pp\-Spin Attractor\):The external thermodynamic drive projects the input requiring𝒪⁡\(B⋅T⋅D⋅Dint\)\\mathcal\{O\}\(B\\cdot T\\cdot D\\cdot D\_\{\\text\{int\}\}\)\. The corepp\-spin attractor iteratesKKmicro\-steps to converge to the fixed point\. Each step executes a linear projection bounded by𝒪⁡\(B⋅T⋅Dint2\)\\mathcal\{O\}\(B\\cdot T\\cdot D\_\{\\text\{int\}\}^\{2\}\)\. Crucially, becauseWextQW\_\{\\text\{ext\}\}^\{Q\}andWintQW\_\{\\text\{int\}\}^\{Q\}are quantized to\{−1,0,1\}\\\{\-1,0,1\\\}, these operations strictly require hardware addition/subtraction, bypassing floating\-point multiplications\. The total time complexity for the equilibrium module per layer is:

𝒪pspin=𝒪⁡\(B⋅T⋅Dint⋅\(D\+K⋅Dint\)\)\\mathcal\{O\}\_\{\\text\{pspin\}\}=\\mathcal\{O\}\\big\(B\\cdot T\\cdot D\_\{\\text\{int\}\}\\cdot\(D\+K\\cdot D\_\{\\text\{int\}\}\)\\big\)\(14\)
3\. Modality Divergence \(Vocabulary Projection\):The sole algorithmic divergence between the Subword and Byte configurations manifests in the final tied embedding projection to the categorical probability simplex, costing𝒪⁡\(B⋅T⋅D⋅\|V\|\)\\mathcal\{O\}\(B\\cdot T\\cdot D\\cdot\|V\|\)\.

- •Subword Modality \(Algorithm 1\):\|V\|=49152\|V\|=49152\. The time complexity is heavily dominated by the output logits computation, introducing a massive𝒪⁡\(B⋅T⋅D⋅49152\)\\mathcal\{O\}\(B\\cdot T\\cdot D\\cdot 49152\)bottleneck\.
- •Byte Modality \(Algorithm 2\):\|V\|=256\|V\|=256\. The projection collapses to a negligible𝒪⁡\(B⋅T⋅D⋅256\)\\mathcal\{O\}\(B\\cdot T\\cdot D\\cdot 256\), making the overall architecture strongly bound by the internal hidden dimensions rather than the vocabulary surface\.

#### 2\.3\.2Space Complexity \(Memory Footprint\)

The spatial demands of the architecture are strictly bounded by two mathematically derived constant\-memory guarantees: SRAM causal chunking and the Implicit Function Theorem \(IFT\)\.

1\. Activation Memory \(Training\):Standard global attention scales quadratically with sequence length𝒪⁡\(B⋅T2⋅H\)\\mathcal\{O\}\(B\\cdot T^\{2\}\\cdot H\)\. By enforcing the Triton\-fused chunked ODE integration, theC×CC\\times Cdecay maskℳ\\mathcal\{M\}is materialized strictly within GPU SRAM and immediately reduced\. Thus, the HBM sequence activation footprint remains strictly linear:𝒪⁡\(B⋅T⋅D\)\\mathcal\{O\}\(B\\cdot T\\cdot D\)\.

Furthermore, standard recurrent fixed point or unrolled solvers require𝒪⁡\(K⋅B⋅T⋅Dint\)\\mathcal\{O\}\(K\\cdot B\\cdot T\\cdot D\_\{\\text\{int\}\}\)memory to store the computational graph for backpropagation\. By utilizing the Neumann series approximation of the inverse Jacobian via the Implicit Function Theorem, the backward pass depends exclusively on the converged stateY∗Y^\{\*\}and intermediate boundsU,GU,Gat stepKK\. The activation memory of thepp\-spin module is unconditionally invariant to the number of micro\-stepsKK:

𝒪act\-pspin=𝒪⁡\(B⋅T⋅Dint\)\\mathcal\{O\}\_\{\\text\{act\-pspin\}\}=\\mathcal\{O\}\(B\\cdot T\\cdot D\_\{\\text\{int\}\}\)\(15\)
2\. Parameter Compression Footprint:The logical parameter matrices are stored as high\-precision weights but deployed as packed representations during execution\. The state\-space matrices \(Wq,Wk,Wv,Wg,WoW\_\{q\},W\_\{k\},W\_\{v\},W\_\{g\},W\_\{o\}\) utilize standard1616\-bit floats\. The internalpp\-spin thermodynamic interactions \(Wext,WintW\_\{\\text\{ext\}\},W\_\{\\text\{int\}\}\) are quantized to1\.581\.58\-bit \(packed into22\-bit blocks\)\. The spatial parameter complexity per layer is:

𝒪params=5⋅16​D2\+2​\(2​D⋅Dint\+2​Dint2\)​bits\\mathcal\{O\}\_\{\\text\{params\}\}=5\\cdot 16D^\{2\}\+2\(2D\\cdot D\_\{\\text\{int\}\}\+2D\_\{\\text\{int\}\}^\{2\}\)\\text\{ bits\}\(16\)This establishes a mathematically rigorous parameter compression ratio, explicitly shrinking the MLP footprint by a theoretical factor of≈8×\\approx 8\\timescompared to equivalent1616\-bit standard architectures\.

3\. Peak Logit VRAM \(Modality Split\):The peak memory allocation during the cross\-entropy gradient formulation is fundamentally bottlenecked by the categorical distribution vector\. For the Subword architecture, this requires𝒪⁡\(B⋅T⋅49152\)\\mathcal\{O\}\(B\\cdot T\\cdot 49152\)floating\-point elements per batch step, dominating the loss calculation tensor space\. Conversely, the Byte\-level architecture bounds this to𝒪⁡\(B⋅T⋅256\)\\mathcal\{O\}\(B\\cdot T\\cdot 256\), allowing extreme context length expansions \(T→∞T\\to\\infty\) given equivalent GPU High\-Bandwidth Memory capacities\.

## 3Experiments

To empirically validate the theoretical properties of thepp\-Spin Glass Network, we evaluate its performance against a heavily optimized standard Transformer baseline\. The evaluation isolates three algorithmic dimensions: sample efficiency, modality agnosticism \(subword vs\. byte level representations\), and gradient stabilization dynamics under extreme batch size constraints\.

### 3\.1Experimental Setup

All models are trained on the HuggingFaceFW/fineweb\-edu dataset \(10BT sample split\)\([11](https://arxiv.org/html/2608.14774#bib.bib21)\)\. The baseline model is a standard autoregressive Transformer optimized with community standard hyperparameters, operating on a batch size ofB=64B=64and trained for1,0001,000steps \(totaling64,00064,000sequences\)\.

The proposedpp\-Spin Glass models are trained in two configurations: a Subword\-level formulation \(\|V\|=49152\|V\|=49152\) and a Byte\-level formulation \(\|V\|=256\|V\|=256\)\. Bothpp\-Spin architectures are trained strictly at a micro\-batch size ofB=1B=1for8,0008,000steps \(totaling8,0008,000sequences\)\. Consequently, the proposed methods are evaluated under an8×8\\timesdata deficit compared to the baseline\. Extensive architectural details, training hyperparameters, and hardware specifications are provided in Appendix[A](https://arxiv.org/html/2608.14774#A1)\.

### 3\.2Sample Efficiency and State\-Space Evaluation

Table[1](https://arxiv.org/html/2608.14774#S3.T1)summarizes the final evaluation metrics\. Despite processing8×8\\timesfewer total sequences, the Subwordpp\-Spin Glass model achieves a final validation cross\-entropy loss of4\.80044\.8004\(Perplexity121\.56121\.56\), marginally outperforming the standard Transformer baseline which converged at4\.80254\.8025\(Perplexity121\.81121\.81\)\.

This demonstrates exceptional sample efficiency\. We attribute this capability to the infinite\-depth continuous routing of the implicit equilibrium forward pass, which contextualizes representations more effectively per stochastic update than standard discrete attention layers\. Furthermore, step execution latency validates the architectural efficiency: the subwordpp\-Spin model executes at0\.440\.44seconds per step \(B=1B=1\), compared to the baseline’s18\.918\.9seconds per step \(B=64B=64\)\.

Table 1:Empirical evaluation of thepp\-Spin Glass models against the Transformer baseline\. Equivalent Subword Loss for the byte\-level model is derived viaLsub≈BPB⋅ln⁡\(2\)⋅3\.8L\_\{\\text\{sub\}\}\\approx\\text\{BPB\}\\cdot\\ln\(2\)\\cdot 3\.8\.![Refer to caption](https://arxiv.org/html/2608.14774v1/loss_vs_sequences.png)

Figure 2:Validation loss as a function of total sequences processed\. Thepp\-Spin model \(Subword\) reaches the baseline’s asymptotic performance utilizing only12\.5%12\.5\\%of the training data\.
### 3\.3Modality Agnosticism: Byte\-Level Processing

Algorithmically, evaluating sequences over raw bytes collapses the VRAM projection footprint by∼192×\\sim 192\\times\(\|V\|=49152\|V\|=49152down to256256\)\. Traditionally, byte\-level autoregressive models suffer from optimization instability due to highly elongated sequence horizons\.

Operating again atB=1B=1for8,0008,000sequences, the Byte\-levelpp\-Spin model reached a final evaluation of2\.122\.12Bits\-Per\-Byte \(BPB\)\. To establish a direct comparison with the subword baseline, we compute the equivalent subword cross\-entropy loss:ℒsub≈BPB⋅ln⁡\(2\)⋅μ\\mathcal\{L\}\_\{\\text\{sub\}\}\\approx\\text\{BPB\}\\cdot\\ln\(2\)\\cdot\\mu, whereμ=3\.8\\mu=3\.8is the empirical bytes per token compression ratio for the baseline tokenizer\. This yields an equivalent subword loss of5\.55\.5\. While this configuration does not surpass the heavily optimized subword baseline under the current computational budget, the monotonically decreasing trend confirms that thepp\-Spin architecture consistently manages temporal credit assignment over uncompressed byte streams\.

### 3\.4Optimization Dynamics and Thermodynamic Stability

A fundamental observation in our experiments is the stability of the optimization trajectory\. In standard deep learning, operating at a batch size of11\(pure Stochastic Gradient Descent\) typically yields a highly chaotic, high\-variance loss surface\. Conversely, thepp\-Spin Glass Network exhibits a perfectly smooth, monotonic convergence curve atB=1B=1, lacking the stochastic jumps characteristic of low\-batch training\.

We formalize this phenomenon asthermodynamic regularization\. The iterativepp\-spin fixed point solver \(Section[2\.1](https://arxiv.org/html/2608.14774#S2.SS1)\) acts as a variance damper\. Because the implicit equilibrium forces the internal representation into a stable energy minimum before calculating the gradient, the backward pass \(computed via the Implicit Function Theorem\) traces an optimized manifold rather than mapping the raw noise of a single input sequence\.

Additionally, because the internal projection matrices \(WintQ,WextQW\_\{\\text\{int\}\}^\{Q\},W\_\{\\text\{ext\}\}^\{Q\}\) are ternarily quantized to\{−1,0,1\}\\\{\-1,0,1\\\}, the Lipschitz constant of the equilibrium layer is strictly bounded\. This prevents catastrophic gradient scaling and eliminates the necessity for massive gradient accumulation, suggesting that thepp\-Spin architecture is uniquely optimal for memory\-constrained continual learning on edge devices\.

![Refer to caption](https://arxiv.org/html/2608.14774v1/variance_dampening.png)

Figure 3:Raw training loss variance ofpp\-Spin Glass model\. Despite utilizing a total batch size of11, thepp\-Spin Glass model exhibits significantly smooth optimization dynamics, showing the variance\-damping properties of the fixed point attractor\.

## 4Conclusion

In this work, we introduced thepp\-Spin Glass Network, which bridges continuous\-time state\-space integration with implicit physical attractors\. The architecture establishes four fundamental advantages over standard baselines:

1. 1\.Memory Efficiency:Native ternary quantization of internal thermodynamic interactions yields an8×8\\timesparameter compression, strictly reducing the byte\-level logical footprint to41\.3741\.37MB\. Additionally, deriving gradients via the Implicit Function Theorem alongside SRAM\-fused chunking bounds the HBM activation memory to𝒪⁡\(B⋅T⋅D\)\\mathcal\{O\}\(B\\cdot T\\cdot D\)\.
2. 2\.Sample Efficiency:The continuous routing of the implicit equilibrium enables asymptotic parity with optimized Transformers \(validation loss of4\.80044\.8004vs\.4\.80254\.8025\), while processing exactly12\.5%12\.5\\%of the training sequences \(8,0008,000vs\.64,00064,000\)\.
3. 3\.Knowledge Acquisition and Continual Learning:The fixed point solver fundamentally dampens gradient variance\. By bounding the Lipschitz constant via ternary quantization, the architecture yields strictly smooth, monotonic convergence at a stochastic micro\-batch size ofB=1B=1, ensuring stable continuous knowledge retention without requiring gradient accumulation\.
4. 4\.Modality Agnosticism:Operating as a general sequence modeling framework, the architecture scales effectively across varying input topologies\. Empirical evaluations confirm robust temporal credit assignment on both heavily compressed subword spaces \(\|V\|=49152\|V\|=49152\) and extreme\-horizon, uncompressed raw byte streams \(\|V\|=256\|V\|=256\)\.

Collectively, the architecture’s native low\-bit compression, high sample efficiency, and single\-batch optimization stability establish a foundation optimal for continuous learning paradigms and memory constrained applications\.

## Appendix AExtended Experimental Details and Parameter Analysis

In this appendix, we provide the complete architectural specifications, optimization hyperparameters, and a breakdown of the memory footprint for both the baseline and the proposedpp\-Spin Glass Sequence architectures\.

### A\.1Dataset and Preprocessing

All models were trained iteratively on thesample\-10BTsplit of theHuggingFaceFW/fineweb\-edudataset\. The training pipeline streams data sequentially with a shuffle buffer size of500500and a predefined random seed \(13371337\)\.

- •Subword Modality:Text is tokenized using theSmolLM2\-135Mtokenizer, yielding a discrete vocabulary space of\|V\|=49152\|V\|=49152\. Sequences are packed to a maximum context length ofT=1024T=1024\.
- •Byte Modality:Text is directly encoded into raw UTF\-8 bytes, strictly bounding the vocabulary space to\|V\|=256\|V\|=256\(with byte255255reserved for EOS\)\. Sequences are packed to a maximum context length ofT=2048T=2048\.

### A\.2Architectural Configurations and Optimization

For fair evaluation, all models were scaled to approximately∼60\\sim 60M logical parameters\. Thepp\-Spin Glass formulations utilizeL=4L=4layers,H=8H=8attention heads, and an equilibrium micro step count ofK=5K=5\. For the State\-Space continuous integration, the chunk size is fixed atC=128C=128\.

To isolate the capabilities of the byte\-level architecture against its subword counterpart and match total parameter number, its internal dimensions were expanded\. Specifically, the Subword model utilizes a hidden dimensionD=544D=544and an intermediate dimensionDint=1408D\_\{\\text\{int\}\}=1408, whereas the Byte model is expanded toD=704D=704andDint=2048D\_\{\\text\{int\}\}=2048\.

Models were trained using AdamW with a weight decay of0\.050\.05and gradient clipping capped at a global norm of1\.01\.0\. The learning rate schedule applied topp\-Spin Glass methods used a linear warmup over the first500500steps peaking at8×10−48\\times 10^\{\-4\}, followed by a stable plateau phase, and concluding with a cosine decay to a minimum ratio of0\.10\.1during the final1,0001,000steps\.

### A\.3Parameter Footprint and Memory Analysis

Table[2](https://arxiv.org/html/2608.14774#A1.T2)and Table[3](https://arxiv.org/html/2608.14774#A1.T3)detail the parameter distribution and logical memory footprint of the models\.

Conversely, thepp\-Spin Glass models achieve a theoretical MLP compression ratio of8\.0×8\.0\\timesby quantizing the internal thermodynamic interactions \(Wext,WintW\_\{\\text\{ext\}\},W\_\{\\text\{int\}\}\) to1\.581\.58\-bit representations \(logically packed as22\-bit structures in hardware\)\. Notably, the Byte\-levelpp\-Spin model heavily reallocates its parameter budget from the vocabulary embedding projection into the core continuous logic matrices \(Ternary MLP and State\-Space Attention\), reducing the global memory footprint to just41\.3741\.37MB while standard transformer or other natively uncompressed model would require \(116\.62116\.62MB\)\.

Table 2:Parameter breakdown for the standard Transformer Baseline\.Table 3:Comprehensive Layer Capacity and Memory Footprint for thepp\-Spin Glass Sequence architectures \(Subword and Byte variations\)\. Natively quantized Ternary matrices strictly achieve an8×8\\timesmemory reduction compared to FP16 baselines\.

## References

- Baiet al\.\(2019\)S\. Bai, J\. Z\. Kolter, and V\. KoltunDeep equilibrium models\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p2.1)\.
- Choromanskaet al\.\(2015\)A\. Choromanska, M\. Henaff, M\. Mathieu, G\. Ben Arous, and Y\. LeCunThe Loss Surfaces of Multilayer Networks\.InProceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics,G\. Lebanon and S\. V\. N\. Vishwanathan \(Eds\.\),Proceedings of Machine Learning Research, Vol\.38,San Diego, California, USA,pp\. 192–204\.External Links:[Link](https://proceedings.mlr.press/v38/choromanska15.html)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p2.1)\.
- Crisanti and Sommers \(1992\)A\. Crisanti and H\. SommersThe spherical p\-spin interaction spin glass model: the statics\.Zeitschrift für Physik B Condensed Matter87\(3\),pp\. 341–354\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. ReFlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=H4DqfPSibmx)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- El Ghaouiet al\.\(2021\)L\. El Ghaoui, F\. Gu, B\. Travacca, A\. Askari, and A\. TsaiImplicit deep learning\.SIAM Journal on Mathematics of Data Science3\(3\),pp\. 930–958\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p2.1)\.
- Gross and Mézard \(1984\)D\. J\. Gross and M\. MézardThe simplest spin glass\.Nuclear Physics B240\(4\),pp\. 431–452\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.
- Gu and Dao \(2023\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.CoRRabs/2312\.00752\.External Links:[Link](https://doi.org/10.48550/arXiv.2312.00752)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Guet al\.\(2022\)A\. Gu, K\. Goel, and C\. ReEfficiently modeling long sequences with structured state spaces\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=uYLFoz1vlAC)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Hopfield \(1982\)J\. J\. HopfieldNeural networks and physical systems with emergent collective computational abilities\.\.Proceedings of the National Academy of Sciences79\(8\),pp\. 2554–2558\.External Links:[Document](https://dx.doi.org/10.1073/pnas.79.8.2554),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.79.8.2554),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.79\.8\.2554Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p2.1)\.
- Khasia \(2026\)V\. KhasiaHoloByte: continuous hyperspherical distillation for tokenizer\-free modeling\.External Links:2603\.16917,[Link](https://arxiv.org/abs/2603.16917)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.
- Lozhkovet al\.\(2024\)A\. Lozhkov, L\. Ben Allal, L\. von Werra, and T\. WolfFineWeb\-edu: the finest collection of educational content\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu),[Document](https://dx.doi.org/10.57967/hf/2497)Cited by:[§3\.1](https://arxiv.org/html/2608.14774#S3.SS1.p1.1)\.
- Maet al\.\(2024\)S\. Ma, H\. Wang, L\. Ma, L\. Wang, W\. Wang, S\. Huang, L\. Dong, R\. Wang, J\. Xue, and F\. WeiThe era of 1\-bit llms: all large language models are in 1\.58 bits\.External Links:2402\.17764,[Link](https://arxiv.org/abs/2402.17764)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Mézardet al\.\(1987\)M\. Mézard, G\. Parisi, and M\. A\. VirasoroSpin glass theory and beyond: an introduction to the replica method and its applications\.Vol\.9,World Scientific Publishing Company\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.
- Parisiet al\.\(2019\)G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. WermterContinual lifelong learning with neural networks: a review\.Neural networks113,pp\. 54–71\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Parisi \(1980\)G\. ParisiThe order parameter for spin glasses: a function on the interval 0\-1\.Journal of Physics A: Mathematical and General13\(3\),pp\. 1101–1112\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Ramsaueret al\.\(2021\)H\. Ramsauer, B\. Schäfl, J\. Lehner, P\. Seidl, M\. Widrich, L\. Gruber, M\. Holzleitner, T\. Adler, D\. Kreil, M\. K\. Kopp, G\. Klambauer, J\. Brandstetter, and S\. HochreiterHopfield networks is all you need\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tL89RnzIiCd)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p2.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Wanget al\.\(2023\)H\. Wang, S\. Ma, L\. Dong, S\. Huang, H\. Wang, L\. Ma, F\. Yang, R\. Wang, Y\. Wu, and F\. WeiBitNet: scaling 1\-bit transformers for large language models\.External Links:2310\.11453,[Link](https://arxiv.org/abs/2310.11453)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p1.1)\.
- Xueet al\.\(2022\)L\. Xue, A\. Barua, N\. Constant, R\. Al\-Rfou, S\. Narang, M\. Kale, A\. Roberts, and C\. RaffelByT5: towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.External Links:[Link](https://aclanthology.org/2022.tacl-1.17/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00461)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.
- YUet al\.\(2023\)L\. YU, D\. Simig, C\. Flaherty, A\. Aghajanyan, L\. Zettlemoyer, and M\. LewisMEGABYTE: predicting million\-byte sequences with multiscale transformers\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=JTmO2V9Xpz)Cited by:[§1](https://arxiv.org/html/2608.14774#S1.p3.1)\.