The Von-Neumann State-Space Transformer for neural decoding

# The Von–Neumann State–Space Transformer for Neural Decoding
Source: [https://arxiv.org/html/2608.25088](https://arxiv.org/html/2608.25088)
Morteza Sarafyazd††thanks:Affiliation: BrainCo, Somerville, US\.

###### Abstract

Cortical computation is strikingly low–dimensional: a handful of latent variables, carried in a neural population’s activity, steer the higher–dimensional responses of individual neurons\. Our aim is sample efficiency—models that decode well from limited data and at small parameter budgets\. In a standard Transformer layer, the feed–forward block applies the same operator to every token\. We suggest a von–Neumann inspired hypothesis of efficient computation as an alternative for neural decoding: a controller decodes an instruction and then executes a token–specific operator; the usual realization—a soft mixture of experts—only blends their outputs, not operators\. We introduce a Von–Neumann State–Space Transformer \(VN–SST\), a memory–augmented Transformer whose feed–forward block is a low–rank instruction bank: a shared base operator plus a small set of learned low–rank instructions, from which a per–token code synthesizes the weight matrix actually used at that token\. The code is read from a low–dimensional projection of a carried state–space memory, so a slow latent trajectory acts as an instruction pointer—mirroring how low–dimensional dynamics may route cortical computation\. On three motor–cortex neural–decoding benchmarks, VN–SST is far more data–efficient than a modern Transformer, each jointly predicting spikes and decoding behavior\. This model wins by a wide margin on the scarcest benchmark, leads on the other two, and turns longer context into rising rather than falling accuracy\. We evaluated that the network compresses a large instruction bank to a few bits per token, so program capacity acts as a control channel, not an accuracy lever\. The same model is also more parameter–efficient on two small text benchmarks used for language modeling \(LLMs\), suggesting a generic mechanism\.

## 1Introduction

Cortical computation is strikingly low–dimensional: population activity spanning thousands of neurons is organized by a handful of latent variables, whose slow trajectories steer the faster, higher–dimensional responses of individual cells\. Our focus is neural decoding that is sample–efficient—accurate behavioral read–out from limited data and small models\. To pursue it we adopt a von–Neumann–inspired hypothesis of efficient computation and build it into a Transformer backbone\. A von–Neumann \(stored–program\) machine has the same structure: a small, slow controller fetches and decodes an instruction, and an execution unit then runs the operator that instruction names, so a compact program steers a large, input–specific computation\. In a standard Transformer, by contrast, the feed–forward network \(FFN\) applies the same fixed weights to every token; we instead give the FFN a controller—a slowly varying state–space memory whose read–out selects, token by token, the operator the FFN executes\.

A natural first attempt is a soft mixture of experts \(soft–MoE\), in which a controller blends the outputs of several fixed FFN experts\[[1](https://arxiv.org/html/2608.25088#bib.bib1)\]\. This is a useful capacity knob, but in the von–Neumann sense it does not execute a program: it mixes the results of fixed operators rather than constructing a token–specific one\. We instead let the controller synthesize the FFN’s weights themselves, per token, from a small shared bank of low–rank “instructions” added to a shared base operator \(Eq\.[7](https://arxiv.org/html/2608.25088#S2.E7), Section[2\.2](https://arxiv.org/html/2608.25088#S2.SS2)\)\. Each token’s code selects a point on a low–dimensional manifold of operators, so the executed map is genuinely token–specific while each added instruction costs little—exposing the size of the instruction bank as a scaling axis of its own, alongside parameters and data, and turning the program into a concrete control signal we can measure\.

The program is generated rather than looked up\. We do not read the instruction off the raw token; instead we condition it on a low–dimensional read–out of a slowly varying selective–state–space \(SSM\) memory carried across the sequence\. The slow latent trajectory then acts as an instruction pointer—low–dimensional dynamics that fetch which operator to run—turning the neuroscience intuition into a mechanism: a small, slow latent drives the higher–complexity, token–specific computation\.

We realize this in VN–SST, a Von–Neumann State–Space Transformer: a drop–in, memory–augmented Transformer—local attention, a selective SSM, and a fast–weight memory, coordinated by a controller—whose feed–forward operator is synthesized per token from the low–rank instruction bank driven by the carried low–dimensional state \(Section[2\.2](https://arxiv.org/html/2608.25088#S2.SS2)\)\. As in a scaling–law study\[[2](https://arxiv.org/html/2608.25088#bib.bib2)\], we compare VN–SST against a modern Transformer on three motor–cortex decoding codecs along three axes—parameters, data, and context \(Sections[4\.1](https://arxiv.org/html/2608.25088#S4.SS1)–[4\.3](https://arxiv.org/html/2608.25088#S4.SS3)\)\. The programmable operator is markedly more sample–efficient: under a limited–data budget it beats the Transformer by a wide margin on the scarcest codec \(decodeR2R^\{2\}0\.350\.35vs\.0\.210\.21\), leads at every data budget on all three, and—uniquely—turns a longer context window into rising rather than falling decode accuracy\. A control–bits diagnostic \(Section[4\.4](https://arxiv.org/html/2608.25088#S4.SS4)\) then makes the von–Neumann claim measurable: the network compresses a3232–instruction bank to only a few bits per token \(≈3\.4\{\\approx\}3\.4–6\.56\.5effective operators\), so program capacity behaves as a control channel, not an accuracy lever—a signal no output–blending mixture can report\.

## 2Models

Both models share a task–agnostic backbone mapping a hidden sequenceh∈ℝB×T×dh\\in\\mathbb\{R\}^\{B\\times T\\times d\}\(batch sizeBB, sequence lengthTT, and hidden widthdd\) to an output of the same shape; only the input/output heads differ\. For the neural codec the input head is a linear map from theNNbinned firing rates and there are two output heads: a linear map back toNNrates \(spike continuation\) and a linear behavioral read–out\.

### 2\.1Transformer baseline

The baseline is a modern decoder–only Transformer\[[3](https://arxiv.org/html/2608.25088#bib.bib3),[4](https://arxiv.org/html/2608.25088#bib.bib4)\]: a stack of identical layers, each combining pre–norm RMSNorm\[[5](https://arxiv.org/html/2608.25088#bib.bib5)\], rotary position embeddings \(RoPE\)\[[6](https://arxiv.org/html/2608.25088#bib.bib6)\], causal multi–head self–attention, and a SwiGLU\[[7](https://arxiv.org/html/2608.25088#bib.bib7)\]feed–forward block\. Every layer applies attention and the feed–forward block as two residual updates,

h←h\+Attn⁡\(RMSNorm⁡\(h\)\),h←h\+SwiGLU⁡\(RMSNorm⁡\(h\)\),h\\leftarrow h\+\\mathrm\{Attn\}\\\!\\big\(\\mathrm\{RMSNorm\}\(h\)\\big\),\\qquad h\\leftarrow h\+\\mathrm\{SwiGLU\}\\\!\\big\(\\mathrm\{RMSNorm\}\(h\)\\big\),\(1\)where the feed–forward block gates one linear projection of its input by another,

SwiGLU⁡\(x\)=W2​\(SiLU⁡\(W1​x\)⊙W3​x\)\.\\mathrm\{SwiGLU\}\(x\)=W\_\{2\}\\big\(\\mathrm\{SiLU\}\(W\_\{1\}x\)\\odot W\_\{3\}x\\big\)\.\(2\)The three weight matricesW1,W3,W2W\_\{1\},W\_\{3\},W\_\{2\}are shared across positions, so the same operator acts on every token\.

### 2\.2VN–SST: memory, state space, and programmable computation

VN–SST keeps the layer skeleton of the baseline but replaces its two blocks: three parallel memory pathways in place of plain attention, and a programmable feed–forward operator in place of the fixed SwiGLU\. These choices are a direct reading of the stored–program hypothesis that motivates this work\. A von–Neumann machine separates a controller that fetches and decodes an instruction, an execution unit that runs the decoded operator, and a memory that persists state between steps; computation is the loop that reads the next instruction and applies it\. VN–SST maps these roles onto a single sequence layer: the persistent selective state–space and fast–weight pathways are the memory, carrying state across tokens; a low–dimensional read–out of the slow state serves as the instruction pointer that selects which operator to run next; the controller decodes that pointer into a per–token instruction code; and the programmable SwiGLU is the execution unit, whose weights that code synthesizes on the fly\. The three pathways and the low–rank instruction bank described below are these roles made concrete\.

Each layer first normalizes its input,u=RMSNorm⁡\(h\)u=\\mathrm\{RMSNorm\}\(h\), and a small controller reads off per–token gates—read–mix gatesgt∈Δ2g\_\{t\}\\in\\Delta^\{2\}\(a simplex weighting over the three pathways\) and memory write gateswtΔ,wtM=σ⁡\(⋅\)w^\{\\Delta\}\_\{t\},w^\{M\}\_\{t\}=\\sigma\(\\cdot\)\. The three pathways then run in parallel\.

#### Pathway 1 — local sensory buffer\.

The first pathway is ordinary multi–head self–attention restricted to a causal band of widthww,

yloc=LocalAttnw​\(u\),y^\{\\text\{loc\}\}=\\mathrm\{LocalAttn\}\_\{w\}\(u\),\(3\)which costs only𝒪⁡\(T​w\)\\mathcal\{O\}\(Tw\)and captures short–range structure inside the current window, but carries nothing across windows—that is the job of the two persistent pathways that follow\.

#### Pathway 2 — selective state space \(slow dynamics\)\.

The second pathway is a diagonal, input–dependent SSM\[[8](https://arxiv.org/html/2608.25088#bib.bib8),[9](https://arxiv.org/html/2608.25088#bib.bib9)\]that carries a per–channel states∈ℝd×ns\\in\\mathbb\{R\}^\{d\\times n\}\(nnlatent states per channel\) across segments\. With input projectionx=Wx​ux=W\_\{x\}u, per–token step sizeΔt=wtΔ​softplus​\(WΔ​ut\+b\)\\Delta\_\{t\}=w^\{\\Delta\}\_\{t\}\\,\\mathrm\{softplus\}\(W\_\{\\Delta\}u\_\{t\}\+b\)\(write gatewtΔw^\{\\Delta\}\_\{t\}; learnedWΔW\_\{\\Delta\}and biasbb\), input–dependent selection vectorsBt,Ct∈ℝnB\_\{t\},C\_\{t\}\\in\\mathbb\{R\}^\{n\}, and diagonal decayA=−exp⁡\(Alog\)A=\-\\exp\(A\_\{\\log\}\)\(learnedAlogA\_\{\\log\}\),

A¯t=exp⁡\(Δt⊙A\),st=A¯t⊙st−1\+\(Δt​xt\)⊗Bt,ytssm=⟨st,Ct⟩\+D⊙xt\.\\displaystyle\\bar\{A\}\_\{t\}=\\exp\(\\Delta\_\{t\}\\odot A\),\\quad s\_\{t\}=\\bar\{A\}\_\{t\}\\odot s\_\{t\-1\}\+\(\\Delta\_\{t\}x\_\{t\}\)\\otimes B\_\{t\},\\quad y^\{\\text\{ssm\}\}\_\{t\}=\\langle s\_\{t\},C\_\{t\}\\rangle\+D\\odot x\_\{t\}\.\(4\)Here⊙\\odotis the elementwise \(Hadamard\) product,⊗\\otimesthe outer product,⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\ranglecontracts over thennstate dimensions, andDDis a learned per–channel skip\. The read–outytssmy^\{\\text\{ssm\}\}\_\{t\}is a low–dimensional projection of the slow statests\_\{t\}; it is exactly the signal we use as the instruction pointer below\.

#### Pathway 3 — fast–weight associative memory \(episodic\)\.

The third pathway is a delta–rule matrix memoryM∈ℝdk×dvM\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{v\}\}\(key and value dimensionsdk,dvd\_\{k\},d\_\{v\}\), also carried across segments, withℓ2\\ell\_\{2\}–normalized keys and querieskt,qt∈ℝdkk\_\{t\},q\_\{t\}\\in\\mathbb\{R\}^\{d\_\{k\}\}and a valuevt∈ℝdvv\_\{t\}\\in\\mathbb\{R\}^\{d\_\{v\}\}\(write gatewtMw^\{M\}\_\{t\}\)\. At each step it writes the current prediction error intoMMand reads content back by query through an output projectionWoW\_\{o\},

Mt=Mt−1\+wtM​kt​\(vt−kt⊤​Mt−1\)⊤,ytmem=Wo​\(qt⊤​Mt\)\.\\displaystyle M\_\{t\}=M\_\{t\-1\}\+w^\{M\}\_\{t\}\\,k\_\{t\}\\big\(v\_\{t\}\-k\_\{t\}^\{\\top\}M\_\{t\-1\}\\big\)^\{\\top\},\\quad y^\{\\text\{mem\}\}\_\{t\}=W\_\{o\}\\big\(q\_\{t\}^\{\\top\}M\_\{t\}\\big\)\.\(5\)Writing the error rather than the raw value makes recall content–addressable: a key that resembles a stored one retrieves its associated value\.

#### Fusion\.

The three read–outs are blended by the controller’s read gates and added back to the residual stream through a linear mapWfW\_\{f\},

rt=gtloc​ytloc\+gtssm​ytssm\+gtmem​ytmem,h←h\+Wf​r\.r\_\{t\}=g\_\{t\}^\{\\text\{loc\}\}y^\{\\text\{loc\}\}\_\{t\}\+g\_\{t\}^\{\\text\{ssm\}\}y^\{\\text\{ssm\}\}\_\{t\}\+g\_\{t\}^\{\\text\{mem\}\}y^\{\\text\{mem\}\}\_\{t\},\\qquad h\\leftarrow h\+W\_\{f\}r\.\(6\)Because the state\(s,M\)\(s,M\)persists across segments, a window of onlywwtokens can carry dependencies far longer thanww\.

#### Programmable compute: the low–rank instruction bank\.

In place of a soft mixture of SwiGLU experts\[[1](https://arxiv.org/html/2608.25088#bib.bib1)\], both projections of the SwiGLU are synthesized per token from shared low–rank banks\. Writing it for a generic projectionWW,

W⁡\(t\)=W0\+∑k=1Kct,k​Uk​Vk⊤,ct∈ℝK,W\(t\)\\;=\\;W\_\{0\}\\;\+\\;\\sum\_\{k=1\}^\{K\}c\_\{t,k\}\\,U\_\{k\}V\_\{k\}^\{\\top\},\\qquad c\_\{t\}\\in\\mathbb\{R\}^\{K\},\(7\)whereW0W\_\{0\}is a shared base operator and\{Uk​Vk⊤\}\\\{U\_\{k\}V\_\{k\}^\{\\top\}\\\}areKKlearned rank–rr“instructions,” selected by the per–token codectc\_\{t\}\. For the input mapWin:ℝd→ℝ2​dffW\_\{\\text\{in\}\}:\\mathbb\{R\}^\{d\}\\\!\\to\\\!\\mathbb\{R\}^\{2d\_\{\\text\{ff\}\}\}\(the concatenated gate/up projection\) the bank hasVk∈ℝd×rV\_\{k\}\\in\\mathbb\{R\}^\{d\\times r\}andUk∈ℝ2​dff×rU\_\{k\}\\in\\mathbb\{R\}^\{2d\_\{\\text\{ff\}\}\\times r\}; for the down mapWdown:ℝdff→ℝdW\_\{\\text\{down\}\}:\\mathbb\{R\}^\{d\_\{\\text\{ff\}\}\}\\\!\\to\\\!\\mathbb\{R\}^\{d\}a second bank hasVkd∈ℝdff×rV^\{d\}\_\{k\}\\in\\mathbb\{R\}^\{d\_\{\\text\{ff\}\}\\times r\}andUkd∈ℝd×rU^\{d\}\_\{k\}\\in\\mathbb\{R\}^\{d\\times r\}, both gated by the same codectc\_\{t\}\. Crucially, no token–specific weight is ever materialized: each programmed projection is a base map plus a low–rank correction,

Win​\(t\)​ut=Win,0​ut\+∑k=1Kct,k​Uk​\(Vk⊤​ut\),W\_\{\\text\{in\}\}\(t\)\\,u\_\{t\}\\;=\\;W\_\{\\text\{in\},0\}\\,u\_\{t\}\\;\+\\;\\sum\_\{k=1\}^\{K\}c\_\{t,k\}\\,U\_\{k\}\\big\(V\_\{k\}^\{\\top\}u\_\{t\}\\big\),\(8\)i\.e\. projectutu\_\{t\}onto theK×rK\{\\times\}rbank, scale blockkkby the codect,kc\_\{t,k\}, and read out throughUU\. Splitting this2​dff2d\_\{\\text\{ff\}\}–dimensional output into twodffd\_\{\\text\{ff\}\}halves\[gate;up\]=Win​\(t\)​ut\[\\,\\text\{gate\};\\text\{up\}\\,\]=W\_\{\\text\{in\}\}\(t\)\\,u\_\{t\}, the hidden activationht=SiLU⁡\(gate\)⊙uph\_\{t\}=\\mathrm\{SiLU\}\(\\text\{gate\}\)\\odot\\text\{up\}is then mapped out through the identically programmedWdown​\(t\)W\_\{\\text\{down\}\}\(t\)\. WithUk,UkdU\_\{k\},U^\{d\}\_\{k\}initialized to zero the layer starts exactly at the shared base SwiGLU\. The per–token cost of the programs is𝒪⁡\(K​r​\(d\+dff\)\)\\mathcal\{O\}\\\!\\big\(Kr\(d\{\+\}d\_\{\\text\{ff\}\}\)\\big\), so program capacityKKscales independently of the base FFN’s𝒪⁡\(d​dff\)\\mathcal\{O\}\(d\\,d\_\{\\text\{ff\}\}\)compute—each added instruction is∼r/dff\\sim\\\!r/d\_\{\\text\{ff\}\}as expensive as a full MoE expert\.

#### The instruction pointer: a manifold–conditioned code\.

The code is decoded from the token and a low–dimensional read–out of the carried SSM state \(Eq\.[4](https://arxiv.org/html/2608.25088#S2.E4)\),

ct=tanh⁡\(MLP⁡\(\[ut;P​ytssm\]\)\)∈\[−1,1\]K,P∈ℝm×d,m≪d,c\_\{t\}\\;=\\;\\tanh\\\!\\Big\(\\mathrm\{MLP\}\\big\(\[\\,u\_\{t\}\\,;\\,P\\,y^\{\\text\{ssm\}\}\_\{t\}\\,\]\\big\)\\Big\)\\;\\in\\;\[\-1,1\]^\{K\},\\qquad P\\in\\mathbb\{R\}^\{m\\times d\},\\ m\\ll d,\(9\)so a slow, low–dimensional latent trajectory \(P​yssmP\\,y^\{\\text\{ssm\}\}\) selects which operator on the manifold executes at each token—the “fetch–decode–execute” loop, with the SSM state as program counter\. Thetanh\\tanhkeeps the synthesized operator on a bounded region of the affine manifold\.

### 2\.3Memory carry and training

Sequences are processed as contiguous segments of lengthLLwith truncated backpropagation through time: the state\(s,M\)\(s,M\)is threaded across segments and detached every few segments; the Transformer runs the identical loop but is stateless \(full attention within each segment\)\. The selective–SSM and delta–rule memories use exact parallel forms \(a log–depth associative scan and a chunkwise WY triangular solve\), so a training step vectorizes on GPU without per–timestep loops\.

## 3Benchmarks and Setup

#### Joint neural codec \(neural sequence generation and behavioral decoding\)\.

We use “codec” in the coder–decoder sense: from one shared hidden state the model must both encode/continue the population spike code and decode behavior\. Concretely, each of three Neural Latents Benchmark\[[10](https://arxiv.org/html/2608.25088#bib.bib10)\]recordings—a standard testbed for latent–dynamics models of motor cortex\[[11](https://arxiv.org/html/2608.25088#bib.bib11)\]—is turned into a dual–objective problem: from the same hidden state the model must \(a\) autoregressively continue the binned population spikes and \(b\) decode behavior—hand or finger velocity—through a second linear head, minimizingℒ=MSEnext–spike\+λ​MSEdecode\\mathcal\{L\}=\\mathrm\{MSE\}\_\{\\text\{next\-\-spike\}\}\+\\lambda\\,\\mathrm\{MSE\}\_\{\\text\{decode\}\}\(λ=10\\lambda\{=\}10, up–weighting the low–dimensional behavioral term against theNN–unit spike term\)\. We use MC\_RTT \(DANDI 000129, M1, random–target reach; finger velocity,N=130N\{=\}130units\), MC\_Maze \(DANDI 000128, M1/PMd, maze reaches; hand velocity,N=182N\{=\}182\), and Area2\_Bump \(DANDI 000127, somatosensory area 2; hand velocity,N=65N\{=\}65\)\. Spikes are binned at50ms50\\text\{\\,\}\\mathrm\{ms\}, smoothed with a Gaussian kernel over33bins, and z–scored; the neural continuation is seeded with a 128–bin prefix\. We report the behavioral–decodingR2R^\{2\}on validation data, with a memoryless ridge decoder \(short causal history\) as a reference\. Neural next–step RMSE stays close to the unit–variance noise floor \(≈1\.0\{\\approx\}1\.0\) and is matched across architectures throughout—single–trial spikes sit near that floor—so decodeR2R^\{2\}is the comparison metric\.

#### Setup\.

For each codec and architecture we build a ladder at fixed depth \(44layers\) and scale only width, targeting the same∼64\{\\sim\}64–270270K non–embedding parameter window for both architectures\. Because VN–SST’s carried state and instruction bank add a fixed per–width overhead, matching this window puts the Transformer on wider blocks and VN–SST on narrower ones, so the two curves overlap on the parameter axis rather than occupying disjoint ranges; fixing depth additionally removes the shape confound that otherwise injects large non–monotonicities into the Transformer’s curve\. The parameter–scaling figure \(Section[4\.1](https://arxiv.org/html/2608.25088#S4.SS1)\) is run at a limited–data budget \(≈25%\{\\approx\}25\\%of the full training budget;∼2\{\\sim\}2/1414/77K bins for MC\_RTT/MC\_Maze/Area2, i\.e\. the smallest budget in Section[4\.2](https://arxiv.org/html/2608.25088#S4.SS2)\), where models sit off the accuracy ceiling; the data– and context–scaling sweeps use the largest available training budget\. All three neural sweeps report means over33seeds; the sequence–length sweep additionally holds the number of optimizer steps fixed across context lengths \(epochs scaled withLL\) so that context length is not confounded with training budget\. Both models use AdamW\[[12](https://arxiv.org/html/2608.25088#bib.bib12)\]with a cosine schedule\[[13](https://arxiv.org/html/2608.25088#bib.bib13)\]and warmup, gradient clipping, identical data budgets and seeds, and5050training epochs\. VN–SST uses windowww, SSM state sizenn, memory dimensionsdk=dvd\_\{k\}\{=\}d\_\{v\}, a manifold read–out dimensionmm, and a bank ofKKrank–rrinstructions as fixed hyperparameters \(defaultK=8K\{=\}8,r=8r\{=\}8,m=8m\{=\}8\)\. We do not match parameter counts; the scaling line reveals efficiency directly\.

## 4Results

We study VN–SST along the three axes a scaling analysis exposes—parameters, data, and training context—always against a modern Transformer under matched budgets\. Throughout, the comparison metric is validation behavioral–decodingR2R^\{2\}; single–trial spike prediction sits near its noise floor for both architectures \(Section[3](https://arxiv.org/html/2608.25088#S3)\), so it is the behavior read–out that separates the models\.

### 4\.1Parameter scaling under limited data

Table[1](https://arxiv.org/html/2608.25088#S4.T1)and Figure[1](https://arxiv.org/html/2608.25088#S4.F1)summarize the parameter sweep at a limited–data budget \(≈25%\{\\approx\}25\\%of the full training budget\)—the scenario single–session recordings actually occupy, and the one in which architecture matters most\. Two things stand out\. First, the parameter ranges are aligned: because VN–SST’s carried state and instruction bank add a large fixed per–width overhead, matching the two architectures on raw parameters requires the Transformer to use wider blocks \(widths4040–8080\) and VN–SST narrower ones \(widths2424–4848\), so both curves cover the same∼64\{\\sim\}64–270270K span rather than sitting on disjoint ranges\. Second, away from the full–data ceiling the curves separate cleanly: replacing the dense FFN with the instruction bank helps most exactly where data is scarce\. On MC\_RTT peak decodeR2R^\{2\}is0\.3510\.351vs\.0\.2070\.207\(mean of33seeds\)—VN–SST beats the Transformer by a wide margin while the memoryless linear decoder is not predictive \(R2=−0\.24R^\{2\}\{=\}\{\-\}0\.24\)—and it keeps a clear edge on the better–sampled Area2\_Bump \(0\.6960\.696vs\.0\.6320\.632\) and MC\_Maze \(0\.7160\.716vs\.0\.6550\.655\) codecs\. Because the instruction bank synthesizes both SwiGLU projections per token, the gain is representational—a richer per–token operator \(Section[2\.2](https://arxiv.org/html/2608.25088#S2.SS2)\)—rather than a parameter effect, and unlike a dense FFN it comes with the measurable control channel of Section[4\.4](https://arxiv.org/html/2608.25088#S4.SS4)\. Because each point is a33–seed mean and the width dependence within a ladder is mild, the robust signal here is the separation between architectures rather than the exact curvature of either curve\.

Table 1:Joint neural codec, parameter scaling with limited data: peak behavioral–decodingR2R^\{2\}\(mean of33seeds\)\. “Linear” is a memoryless ridge decoder with short causal history\.\(a\)MC\_RTT \(finger vel\.\)
\(b\)MC\_Maze \(hand vel\.\)
\(c\)Area2\_Bump \(hand vel\.\)

Figure 1:Neural–codec parameter scaling with limited data\. \(a–c\) Behavioral–decodingR2R^\{2\}vs\. non–embedding parameters on the MC\_RTT, MC\_Maze, and Area2\_Bump codecs, respectively, for the Transformer \(blue\) and VN–SST \(green, “instruction bank”\), with a linear reference\. The two ladders are aligned to the same∼64\{\\sim\}64–270270K window \(Transformer on widths4040–8080, VN–SST on2424–4848\), since VN–SST’s carried state and instruction bank add a fixed per–width overhead\. Curves show the mean over33seeds\. Away from the accuracy ceiling the architectures separate, with VN–SST leading on all three codecs; the margin is largest on the data–scarce MC\_RTT codec in \(a\), where VN–SST reachesR2=0\.35R^\{2\}\{=\}0\.35against0\.210\.21for the Transformer and the memoryless linear decoder is not predictive \(R2<0R^\{2\}\{<\}0\)\.
### 4\.2Data scaling

We fix the model \(the fixed–depth∼150\{\\sim\}150K decoding model\) and vary the amount of training data over four recording–bin budgets per codec\.

Table 2:Data scaling at the fixed–depth∼150\{\\sim\}150K decoding model \(5050epochs, smoothed rates, mean of33seeds\): decodeR2R^\{2\}at the smallest→\\tolargest training budget \(four recording–bin budgets per codec\)\.\(a\)MC\_RTT codec
\(b\)MC\_Maze codec
\(c\)Area2\_Bump codec

Figure 2:Data scaling at the fixed–depth∼150\{\\sim\}150K decoding model \(mean of33seeds\)\. \(a–c\) DecodeR2R^\{2\}vs\. training bins on the MC\_RTT, MC\_Maze, and Area2\_Bump codecs, respectively, for the Transformer \(blue\) and VN–SST \(green\)\.Across the four data budgets VN–SST is at least as accurate as the Transformer on all three codecs, and its advantage is largest where data is scarcest\. At the smallest budget it is ahead on Area2\_Bump \(decodeR2R^\{2\}0\.680\.68vs\.0\.600\.60at∼7\{\\sim\}7k bins\) and on MC\_RTT \(0\.300\.30vs\.0\.190\.19at∼2\{\\sim\}2k bins\); the gap then narrows as data grows, with the two models close at the full recording \(MC\_RTT0\.610\.61vs\.0\.520\.52; MC\_Maze0\.850\.85vs\.0\.850\.85; Area20\.790\.79vs\.0\.780\.78\)\. A plausible reading is that the carried low–dimensional state supplies temporal context that partly substitutes for data, so the clearest gains appear under the tightest budgets and diminish as data grows\.

### 4\.3Sequence–length scaling

Holding the model and the full training set fixed, we varied only the truncated–BPTT segment lengthL∈\{16,32,64,128\}L\\in\\\{16,32,64,128\\\}\. Because the number of optimizer steps per epoch scales asT/LT/L, a fixed–epoch sweep would confound context length with training budget: longer segments yield proportionally fewer updates\. We therefore equalized the number of optimizer steps across conditions by scaling the epoch count withLL, and we report the mean over33seeds\.

Table 3:Sequence–length scaling at the fixed–depth∼150\{\\sim\}150K decoding model and full data, at a matched training budget \(epochs scaled withLLso the number of optimizer steps is held constant across context lengths; mean of33seeds\): best decodeR2R^\{2\}overL∈\{16,32,64,128\}L\\in\\\{16,32,64,128\\\}and theLLat which it peaks \(each cell reads bestR2R^\{2\}@LL\)\. On the scarce MC\_RTT codec the two diverge—the Transformer’s decode gently declines asLLgrows \(→0\.570\.61\\\!\\to\\\!0\.57\) while VN–SST’s rises \(→0\.680\.60\\\!\\to\\\!0\.68\), converting longer context into accuracy through its carried state; on the data–rich codecs both plateau \(VN–SST slightly higher and flatter, and it alone holds up atL=128L\{=\}128\)\.\(a\)MC\_RTT codec
\(b\)MC\_Maze codec
\(c\)Area2\_Bump codec

Figure 3:Sequence–length scaling at fixed model and full data, at a matched update budget \(epochs scaled withLL; mean of33seeds\): decodeR2R^\{2\}vs\. training context lengthLL\(log2xx–axis\), on the MC\_RTT \(a\), MC\_Maze \(b\), and Area2\_Bump \(c\) codecs\. \(a\) On the scarce MC\_RTT codec the two architectures cross over: the Transformer starts higher atL=16L\{=\}16but declines as the window grows \(→0\.570\.61\\\!\\to\\\!0\.57\), whereas VN–SST rises \(→0\.680\.60\\\!\\to\\\!0\.68\) and overtakes it—threading state across segments, it converts longer context into decoding accuracy while the Transformer’s wider receptive field does not help its short–horizon behavioral read–out\. \(b,c\) On the data–rich MC\_Maze and Area2\_Bump codecs both are high and largely flat, with VN–SST slightly higher\.The context axis mirrors the data axis: longer context matters most where data is the binding constraint, and it is where the two architectures behave most differently\. Once the update budget is matched—removing the fixed–epoch confound that would otherwise exaggerate any trend—the scarce MC\_RTT codec shows a clean crossover\. The Transformer starts ahead atL=16L\{=\}16\(0\.610\.61\) but its decode drifts down as the window grows\. This is expected for the task rather than a general context effect: instantaneous velocity is a short–horizon readout of the current population state, so with the update budget already matched a longer window carries little additional predictive signal\. What it does add is a wider attention receptive field \(more capacity to overfit\) and, since a fixed recording yieldsT/LT/Lsegments, fewer independent training sequences per epoch—both of which slightly hurt generalization for a plain Transformer, unlike language modeling, where the target genuinely depends on long–range context\. VN–SST instead rises monotonically and overtakes it byL≈24L\{\\approx\}24\. One reason is likely that its carried low–dimensional state integrates the extra context for the read–out without widening the local window\. On the data–rich codecs both stay high and largely flat \(MC\_Maze≈0\.87\{\\approx\}0\.87, Area2≈0\.77\{\\approx\}0\.77; Table[3](https://arxiv.org/html/2608.25088#S4.T3)\), but VN–SST is consistently slightly higher\.

### 4\.4The program manifold and control bits

Unlike a dense FFN or an output–blending MoE, the instruction bank has an explicit capacity knob—the program–manifold dimensionKK\. At a fixed model size \(so per–token compute barely moves\) we sweepK∈\{1,…,32\}K\\in\\\{1,\\dots,32\\\}and, for every run, both measure decodeR2R^\{2\}and collect the per–token codesctc\_\{t\}on validation data to quantify how much of the program space the network actually uses\. We report two summaries of the empirical code distribution: the spectral entropy of its covariance \(“code bits,” bits of instruction variation\) and the participation ratio \(the effective number of instructions used\)\. Table[4](https://arxiv.org/html/2608.25088#S4.T4)and Figure[4](https://arxiv.org/html/2608.25088#S4.F4)report the result\.

Table 4:Control–bits diagnostic\. Given a program bank of sizeK=32K\{=\}32at a fixed model size, we collect the per–token instruction codesctc\_\{t\}on validation data and report the spectral entropy of their covariance \(bits of code variation\) and the participation ratio \(effective number of instructions used\), alongside the range of decodeR2R^\{2\}overK∈\{1,…,32\}K\\in\\\{1,\\dots,32\\\}\. On every codec the model compresses a3232–instruction bank to≈2\.5\{\\approx\}2\.5–3\.23\.2bits /≈3\.4\{\\approx\}3\.4–6\.56\.5effective operators—well below thelog2⁡K=5\\log\_\{2\}K\{=\}5ceiling—while decodeR2R^\{2\}is essentially flat inKK: program capacity is a control/compression knob, not an accuracy lever\.\(a\)decodeR2R^\{2\}vs\.KK\(MC\_RTT\)
\(b\)control bits \(MC\_RTT\)
\(c\)control bits \(Area2\)

Figure 4:Program–manifold sweep at fixed model size\. \(a\) DecodeR2R^\{2\}is essentially flat and non–monotonic in the program countKK—capacity is not the bottleneck here\. \(b,c\) The per–token instruction code compresses: givenK=32K\{=\}32instructions the network uses only≈2\.5\{\\approx\}2\.5–3\.23\.2bits of code entropy and a participation ratio of≈3\.4\{\\approx\}3\.4–6\.56\.5, bending progressively below thelog2⁡K\\log\_\{2\}Kceiling—so the network uses only a few instructions, consistently across recordings\.In this study, two observations emerge, and they are the point of the architecture\. \(1\) Program capacity is not an accuracy lever here\. DecodeR2R^\{2\}is essentially flat and non–monotonic acrossKK\(Fig\.[4](https://arxiv.org/html/2608.25088#S4.F4)a\): the small motor codes these recordings support are already captured by a handful of operators, so adding instructions neither helps nor hurts\. \(2\) The network compresses to a small instruction set\. Given a bank ofK=32K\{=\}32, the empirical code uses only≈2\.55\{\\approx\}2\.55bits \(MC\_RTT\),3\.203\.20\(Area2\), and2\.892\.89\(MC\_Maze\)—against a ceiling oflog2⁡32=5\\log\_\{2\}32=5—with participation ratios of≈3\.4\{\\approx\}3\.4–6\.56\.5effective instructions \(Table[4](https://arxiv.org/html/2608.25088#S4.T4)\)\. The entropy curve bends below the ceiling asKKgrows \(Fig\.[4](https://arxiv.org/html/2608.25088#S4.F4)b,c\): the model does not spread across all available programs but concentrates on a few, and it does so consistently across three brain areas\. This is the von–Neumann claim made numeric—the layer runs on≈2\.5\{\\approx\}2\.5–33bits of program per token—and it is a property only operator synthesis can report\. An output–blending MoE has no addressable instruction whose entropy can be measured\. It also suggests a design rule and an interpretability handle \(which few operators specialize, and how they compose\) that we leave to future work\.

### 4\.5Additional modality check: language

Although this paper is about neural decoding, the instruction bank is a generic feed–forward mechanism, so it is worth asking whether it also helps on a very different sequence modality\. We therefore additionally tested the same VN–SST \(identical bank sizeKK, rankrr, and manifold read–out\) against the Transformer on two sub–word \(byte–level BPE\) text corpora, tiny–Shakespeare and WikiText–2, over a fixed–depth, width–scaled ladder \(two layers;∼0\.13\{\\sim\}0\.13–1\.71\.7M non–embedding parameters; mean of33seeds\), matching the fixed–depth protocol used for the neural codecs\. To see how the comparison moves with data, we run each corpus at two budgets: a small “1×\\times” budget and a “3×\\times” budget with three times as many training and validation tokens \(tiny–Shakespeare114114K→342\{\\to\}342K, its full corpus; WikiText–2440440K→1\.32\{\\to\}1\.32M\)\.

Table[5](https://arxiv.org/html/2608.25088#S4.T5)and Figure[5](https://arxiv.org/html/2608.25088#S4.F5)show that the instruction bank transfers, and that the effect is robust to data scale\. At both budgets and on both corpora, VN–SST tracks below the Transformer across the ladder and has the steeper power–law slope, so it is more parameter–efficient throughout\. Adding data helps both models—perplexity falls sharply from the 1×\\timesto the 3×\\timesbudget \(WikiText–2 top–of–ladder→52\.195\.7\\\!\\to\\\!52\.1for the Transformer and→45\.174\.4\\\!\\to\\\!45\.1for VN–SST; tiny–Shakespeare→60\.6174\.8\\\!\\to\\\!60\.6and→49\.4151\.2\\\!\\to\\\!49\.4\)—but it does not erase the gap\. At the largest budget VN–SST still attains the best perplexity on each corpus,49\.449\.4vs\.60\.660\.6on tiny–Shakespeare and45\.145\.1vs\.52\.152\.1on WikiText–2 \(a≈13\{\\approx\}13–18%18\\%reduction at the top of the ladder\), and it reaches the Transformer’s best with roughly22–3×3\\timesfewer parameters\. The one nuance is how the lead moves with scale: on tiny–Shakespeare it widens with more data \(VN–SST’s slope steepens to−0\.33\-0\.33\), whereas on WikiText–2 the slopes flatten as both models saturate and the margin narrows, though VN–SST stays ahead\. We read this as evidence that per–token operator synthesis is a generic gain—a programmable operator plus the state and fast–weight pathways help wherever long–range structure must be integrated from a small window—rather than a quirk of the neural codecs\. We keep the study framed around neural decoding, where the small–data, small–model regime makes the memory and program machinery most decisive, and treat language as a check on generality\.

Table 5:Additional modality check—sub–word language modeling\. Validation perplexity \(best over a∼0\.13\{\\sim\}0\.13–1\.71\.7M non–embedding parameter ladder at fixed two–layer depth with width scaled, mean of33seeds, at the largest \(3×3\\times\) data budget\) and the fitted power–law slope vs\. parameters, for the Transformer and VN–SST on byte–level BPE tiny–Shakespeare and WikiText–2\. The instruction–bank model, unchanged from the neural experiments, is more parameter–efficient across the ladder on both corpora, attains the best perplexity on both, and has the steeper slope on both\.\(a\)tiny–Shakespeare, 1×\\times\(114114K tokens\)
\(b\)WikiText–2, 1×\\times\(440440K tokens\)
\(c\)tiny–Shakespeare, 3×\\times\(342342K tokens\)
\(d\)WikiText–2, 3×\\times\(1\.321\.32M tokens\)

Figure 5:Additional modality check: language–model scaling at two data budgets\. Validation perplexity vs\. non–embedding parameters \(log–log, with power–law fits\) for the Transformer and VN–SST\. Top row \(a,b\): the small “1×\\times” budget; bottom row \(c,d\): the “3×\\times” budget with three times as many tokens; tiny–Shakespeare in \(a,c\) and WikiText–2 in \(b,d\)\. The instruction–bank model—unchanged from the neural experiments—tracks below the Transformer across the ladder in every panel, has the steeper slope, and reaches the best perplexity\. More data lowers both curves; VN–SST’s lead widens with data on tiny–Shakespeare and narrows on WikiText–2 \(where both models saturate\), but it stays ahead throughout\.

## 5Conclusion

We replaced a Transformer’s fixed feed–forward operator with a low–rank instruction bank—a shared base operator plusKKrank–rrinstructions—from which a per–token, manifold–conditioned code synthesizes the operator that the layer actually executes\. The code is read from a carried low–dimensional state, so a slow latent trajectory—the signal that organizes cortical population activity—acts as an instruction pointer\. This makes the von–Neumann “programmable FFN” concrete: rather than blending the outputs of a few fixed experts, the layer constructs a token–specific operator on a low–dimensional weight manifold\.

The mechanism earns its keep exactly where real neural recordings live—small data and small models\. On three motor–cortex decoding codecs VN–SST matches or beats a modern Transformer along all three scaling axes: under a limited–data budget it leads by a wide margin on the scarcest codec, it is at least as accurate at every data budget on all three, and—uniquely—it turns a longer context window into rising rather than falling decode accuracy\. And because the operator is addressed rather than blended, the control signal itself is measurable: the network compresses a3232–instruction bank to only≈2\.5\{\\approx\}2\.5–3\.23\.2bits \(≈3\.4\{\\approx\}3\.4–6\.56\.5effective operators\) per token, while decode accuracy stays flat inKK\. Program capacity is therefore a control channel, not an accuracy lever\.

More broadly, these results suggest that when data and models are scarce, the useful inductive bias is not simply more parameters but a programmable operator steered by low–dimensional dynamics—computation that, like the cortex it models, runs on a few well–chosen instructions\. Two pieces are reusable beyond neural decoding: a parameter–efficient way to make feed–forward compute token–programmable, and a way to quantify how much program a trained model actually uses\.

## Code and data availability

Code to reproduce all experiments, figures, and tables—including the model implementations, training loop, and scaling sweeps—is available from the authors upon request, subject to internal review\. The neural benchmarks are the public Neural Latents Benchmark\[[10](https://arxiv.org/html/2608.25088#bib.bib10)\]recordings on DANDI \(dandisets 000127, 000128, and 000129\); the text corpora are tiny–Shakespeare and WikiText–2, both publicly available\.

## Acknowledgments

We are grateful for our institution’s support of this study\. We thank the Neural Latents Benchmark and DANDI teams for curating and publicly hosting the datasets used in this work\. This work is intended for scholarly purposes\.

## References

- \[1\]N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\. Outrageously large neural networks: The sparsely–gated mixture–of–experts layer\.*International Conference on Learning Representations \(ICLR\)*, 2017\. arXiv:1701\.06538\.
- \[2\]J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei\. Scaling laws for neural language models\.*arXiv:2001\.08361*, 2020\.
- \[3\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\. Attention is all you need\.*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2017\. arXiv:1706\.03762\.
- \[4\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\.\-A\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample\. LLaMA: Open and efficient foundation language models\.*arXiv:2302\.13971*, 2023\.
- \[5\]B\. Zhang and R\. Sennrich\. Root mean square layer normalization\.*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2019\. arXiv:1910\.07467\.
- \[6\]J\. Su, Y\. Lu, S\. Pan, B\. Wen, and Y\. Liu\. RoFormer: Enhanced transformer with rotary position embedding\.*arXiv:2104\.09864*, 2021\.
- \[7\]N\. Shazeer\. GLU variants improve transformer\.*arXiv:2002\.05202*, 2020\.
- \[8\]A\. Gu, K\. Goel, and C\. Ré\. Efficiently modeling long sequences with structured state spaces\.*International Conference on Learning Representations \(ICLR\)*, 2022\. arXiv:2111\.00396\.
- \[9\]A\. Gu and T\. Dao\. Mamba: Linear–time sequence modeling with selective state spaces\.*arXiv:2312\.00752*, 2023\.
- \[10\]F\. Pei, J\. Ye, D\. Zoltowski, A\. Wu, R\. H\. Chowdhury, H\. Sohn, J\. E\. O’Doherty, K\. V\. Shenoy, M\. T\. Kaufman, M\. Churchland, M\. Jazayeri, L\. E\. Miller, J\. Pillow, I\. M\. Park, E\. L\. Dyer, and C\. Pandarinath\. Neural Latents Benchmark ’21: Evaluating latent variable models of neural population activity\.*Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track*, 2021\. arXiv:2109\.04463\.
- \[11\]C\. Pandarinath, D\. J\. O’Shea, J\. Collins, R\. Jozefowicz, S\. D\. Stavisky, J\. C\. Kao, E\. M\. Trautmann, M\. T\. Kaufman, S\. I\. Ryu, L\. R\. Hochberg, J\. M\. Henderson, K\. V\. Shenoy, L\. F\. Abbott, and D\. Sussillo\. Inferring single–trial neural population dynamics using sequential auto–encoders\.*Nature Methods*, 15:805–815, 2018\. doi:10\.1038/s41592–018–0109–9\.
- \[12\]I\. Loshchilov and F\. Hutter\. Decoupled weight decay regularization\.*International Conference on Learning Representations \(ICLR\)*, 2019\. arXiv:1711\.05101\.
- \[13\]I\. Loshchilov and F\. Hutter\. SGDR: Stochastic gradient descent with warm restarts\.*International Conference on Learning Representations \(ICLR\)*, 2017\. arXiv:1608\.03983\.