Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memories

# Subspace Carving in Order-𝑝 Tensor Memories
Source: [https://arxiv.org/html/2606.11391](https://arxiv.org/html/2606.11391)
## Recursive Binding on a Budget: Subspace Carving in Order\-ppTensor Memories

###### Abstract

Tensor Product Representations provide the structural fidelity required for symbolic reasoning in models but suffer fromexponentialdimensionality growth when encoding deep recursive structures\. Conversely, Vector Symbolic Architectures maintainconstantdimensionality but sacrifice capacity and fidelity due to noisy compression via superposition\. In this work, we proposeOrthogonal Subspace Carving \(OSC\), a memory architecture that bindsfillerstorolesby projecting onto the null space of the role basis before aggregating into a fixed order\-pptensor\. OSC uses projections to enforce geometric orthogonality between bound structures within astaticmemory trace\. We show that this mechanism decouples the tensor order from the structural depth, enabling deep recursive binding within aconstantmemory footprint\. By performing retrieval via recognition, this construction allows for component vectors that areorders of magnitudesmaller than the memory tensor, giving superior memory efficiency in settings involving high superposition\. We also show that TPR is a special case of binding in Clifford algebra, and give a Clifford formulation of OSC\.

Machine Learning, ICML

## 1Introduction

Human reasoning is distinguished by its ability to perform structured operations over symbolic representations\(Newell,[1980](https://arxiv.org/html/2606.11391#bib.bib1),[1982](https://arxiv.org/html/2606.11391#bib.bib2); Marcus,[2001](https://arxiv.org/html/2606.11391#bib.bib4)\)\. Standard neural network models struggle to systematically generalize to novel compositions of known structures\(Dziriet al\.,[2023](https://arxiv.org/html/2606.11391#bib.bib8); Liet al\.,[2023](https://arxiv.org/html/2606.11391#bib.bib7); Kim and Linzen,[2020](https://arxiv.org/html/2606.11391#bib.bib6); Keyserset al\.,[2020](https://arxiv.org/html/2606.11391#bib.bib5)\)\. Bridging this “neuro\-symbolic gap” requires representations that can support rich, recursive structure but are compatible with the gradient\-based optimization of continuous vector spaces\(Kanerva,[2009](https://arxiv.org/html/2606.11391#bib.bib3); Kleykoet al\.,[2022](https://arxiv.org/html/2606.11391#bib.bib15)\)\.

An important line of work addressing the challenge above is the Tensor Product Representation \(TPR\)\(Smolensky,[1990](https://arxiv.org/html/2606.11391#bib.bib9)\)\. As we will discuss in more detail shortly, TPRs provide a mathematically rigorous method for binding variables \(“fillers”\) to structural “roles” via the outer product operation\. This explicitly constructs a representation that preserves perfect orthogonality between bindings, and allows exact retrieval of constituents without interference \(i\.e\., perfectly\)\. However, the dimensionality of a TPR scalesexponentiallywith the depth of the structure\. For example, a tree of depthkkneeds a tensor space of orderdkd^\{k\}, making exact deep recursive binding intractable\(Souloset al\.,[2023](https://arxiv.org/html/2606.11391#bib.bib10)\)\. While recent sparse approximations can mitigate this scaling, they suffer from significant performance degradation on deep structures\(Souloset al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib11)\)\.

Vector Symbolic Architectures \(VSAs\) offer a compressed alternative\(Kanerva,[2009](https://arxiv.org/html/2606.11391#bib.bib3)\)\. VSAs employ “reduced” binding operations, such as circular convolution or element\-wise multiplication, that map the tensor product back into the fixed\-dimensional space of the component vectors\(Kleykoet al\.,[2022](https://arxiv.org/html/2606.11391#bib.bib15)\)\. This fixed dimensionality makes VSAs efficient\. It allows encoding complex structures within aconstantmemory footprint\. But, there is a trade\-off\. In VSAs that use approximate inverses, non\-orthogonal noiseaccumulateswith depth\. Further, as memory becomes crowded \(more superposition\), SNR degrades, eventually impacting reliable retrieval\(Gosmann and Eliasmith,[2019](https://arxiv.org/html/2606.11391#bib.bib12); Plate,[1995](https://arxiv.org/html/2606.11391#bib.bib13)\)\.

Ourcontributionsare:\(a\)OSC, a novel memory architecture that decouples component vector dimension from memory capacity via a projection\-based binding mechanism;\(b\)a large reduction in memory footprint for high\-superposition tasks\. We get robust retrieval with component vectors orders of magnitude smaller than the memory space;\(c\)near\-lossless retrieval accuracy in high\-superposition regimes, enabled by sub\-linear filler scaling that allows for larger superposition memory spaces within a smaller total memory footprint without TPR’s exponential dimensionality cost\. We do extensive empirical verification against VSA and in standard downstream applications\. The code is available at[https://github\.com/vsingh\-group/OrthogonalSubspaceCarving](https://github.com/vsingh-group/OrthogonalSubspaceCarving)\.

Conflict of Interest Disclosure\.There are no conflicts of interest\.

## 2Preliminaries

We first give a short overview of Tensor Product Representations \(TPRs\) and Vector Symbolic Architectures \(VSAs\)\.

Spaces\.VSAs/TPRs operate with three specific types of “spaces”, namely,filler,role, andmemory\. Afillerrepresents the atomic content, concept, or object being stored \(e\.g\., “blue”, “dog”, “5”\)\. Aroledefines the structural slot or attribute associated with that content \(e\.g\., “color”, “animal”, “magnitude”\)\. Theassociationof a role\(s\) to a filler creates an instance of that concept within the memory, allowing hierarchical data to be represented\. See[Figure1](https://arxiv.org/html/2606.11391#S2.F1)\.

###### Definition 2\.1\(VSA/TPR Filler\)\.

Thefiller spaceis the hyperspace𝒱\\mathcal\{V\}, usuallyℝd\\mathbb\{R\}^\{d\},ℂd\\mathbb\{C\}^\{d\}, or\{0,1\}d\\\{0,1\\\}^\{d\}\. A specificfillerf∈𝒱f\\in\\mathcal\{V\}is a single vector in the filler space\.

###### Definition 2\.2\(VSA/TPR Context\)\.

Similarly, therole spaceis also𝒱\\mathcal\{V\}\. A specificroler∈𝒱r\\in\\mathcal\{V\}is a single vector\. Acontext𝒞\\mathcal\{C\}is just a collection of rolesr1,r2,…,rc∈𝒱r\_\{1\},r\_\{2\},\\ldots,r\_\{c\}\\in\\mathcal\{V\}\.

VSAs and TPRs use three main manipulation operations\.

1. 1\.Binding \(⊗\\otimes\):This operatorassociatesthe role with the filler, creating the bound objectT=f⊗rT=f\\otimes r\.
2. 2\.Unbinding \(⊘\\oslash\):This operator uses the role toretrievethe filler from the object,\(f⊗r\)⊘r=f\(f\\otimes r\)\\oslash r=f\.
3. 3\.Bundling \(⊕\\oplus\):This operator superposes multiple bound objects into one memory space,M=T1⊕T2⊕⋯⊕TkM=T\_\{1\}\\oplus T\_\{2\}\\oplus\\dots\\oplus T\_\{k\}\. This is almost always simple addition\.

𝟓\\boldsymbol\{5\}𝝅\\boldsymbol\{\\pi\}Step 1Codebook\[amp\] =rampr\_\{\{\\color\[rgb\]\{0\.7,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0,0\}\\textbf\{amp\}\}\}\[freq\] =rfreqr\_\{\{\\color\[rgb\]\{0,0\.6,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.6,0\}\\textbf\{freq\}\}\}\[5\] =f𝟓f\_\{\{\\color\[rgb\]\{0,0,0\.7\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\.7\}\\boldsymbol\{5\}\}\}\[𝝅\\boldsymbol\{\\pi\}\] =f𝝅f\_\{\{\\color\[rgb\]\{0\.7,0\.35,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0\.35,0\}\\boldsymbol\{\\pi\}\}\}Step 2rampr\_\{\{\\color\[rgb\]\{0\.7,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0,0\}\\textbf\{amp\}\}\}⊗\\otimesf𝟓f\_\{\{\\color\[rgb\]\{0,0,0\.7\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\.7\}\\boldsymbol\{5\}\}\}==ramp⊗f𝟓r\_\{\{\\color\[rgb\]\{0\.7,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0,0\}\\textbf\{amp\}\}\}\\otimes f\_\{\{\\color\[rgb\]\{0,0,0\.7\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\.7\}\\boldsymbol\{5\}\}\}rfreqr\_\{\{\\color\[rgb\]\{0,0\.6,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.6,0\}\\textbf\{freq\}\}\}⊗\\otimesf𝝅f\_\{\{\\color\[rgb\]\{0\.7,0\.35,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0\.35,0\}\\boldsymbol\{\\pi\}\}\}==rfreq⊗f𝝅r\_\{\{\\color\[rgb\]\{0,0\.6,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.6,0\}\\textbf\{freq\}\}\}\\otimes f\_\{\{\\color\[rgb\]\{0\.7,0\.35,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0\.35,0\}\\boldsymbol\{\\pi\}\}\}⊕\\oplusMemory\(ramp⊗f𝟓\)⊕\(rfreq⊗f𝝅\)\\begin\{aligned\} \(r\_\{\{\\color\[rgb\]\{0\.7,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0,0\}\\textbf\{amp\}\}\}\\otimes&f\_\{\{\\color\[rgb\]\{0,0,0\.7\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,0\.7\}\\boldsymbol\{5\}\}\}\)\\\\ \\oplus&\\\\ \(r\_\{\{\\color\[rgb\]\{0,0\.6,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.6,0\}\\textbf\{freq\}\}\}\\otimes&f\_\{\{\\color\[rgb\]\{0\.7,0\.35,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.7,0\.35,0\}\\boldsymbol\{\\pi\}\}\}\)\\end\{aligned\}Step 3

Figure 1:Raw signal attributes are identified \(Step 1\) and mapped to vectors using a codebook \(Step 2\)\. These are then bound to their corresponding roles and superposed to form the memory \(Step 3\)\.### 2\.1Tensor Product Representations \(TPRs\)

In TPRs, the memory space isnotthe same as the role and filler space but instead is a higher\-order tensor\.

###### Definition 2\.3\(TPR Binding\)\.

The binding operator is simply the tensor product\. Given a fillerf∈ℝdf\\in\\mathbb\{R\}^\{d\}and a roler∈ℝdr\\in\\mathbb\{R\}^\{d\}, the bound objectTTis a matrix inℝd×d\\mathbb\{R\}^\{d\\times d\}:

T=f⊗r=f​r⊤T=f\\otimes r=fr^\{\\top\}\(1\)

So, when we associate11role to a filler, the memory space is an order\-22tensor\. When binding\|𝒞\|\|\\mathcal\{C\}\|roles to a filler, the memory space is a order\-\(\|𝒞\|\+1\)\|\\mathcal\{C\}\|\+1\)tensor withd\|𝒞\|\+1d^\{\|\\mathcal\{C\}\|\+1\}entries\. This grows exponentially with context size\.

###### Definition 2\.4\(TPR Unbinding\)\.

Unbinding is the inner productcontractionof the bound object with the role vector:

Orthogonality and Superposition\.A key property of TPRs is that if the role vectors are mutually orthogonal, superposition islossless\. That is, multiple filler\-role bindings can be bundled into a single memory matrixMMwithoutany interferencebetween terms\. Let rolesr1,r2r\_\{1\},r\_\{2\}be orthogonal\.

###### Example 2\.5\.

For memoryM=f1​r1⊤\+f2​r2⊤M=f\_\{1\}r\_\{1\}^\{\\top\}\+f\_\{2\}r\_\{2\}^\{\\top\}, retrieval is exact:

M​r1=f1​\(r1⊤​r1\)\+f2​\(r2⊤​r1\)=f1​\(1\)\+f2​\(0\)=f1Mr\_\{1\}=f\_\{1\}\(r\_\{1\}^\{\\top\}r\_\{1\}\)\+f\_\{2\}\(r\_\{2\}^\{\\top\}r\_\{1\}\)=f\_\{1\}\(1\)\+f\_\{2\}\(0\)=f\_\{1\}\(3\)

### 2\.2Vector Symbolic Architectures \(VSAs\)

TPRs guarantee perfect retrieval via orthogonality but memory size scales exponentially \(with context size\)\. VSAs instead use compressed, dimensionality\-preserving operations\. Thus, VSAs can be viewed as an approximation of TPRs\. Basically, the tensor product is “collapsed” back into the original vector space𝒱\\mathcal\{V\}, trading exactness for a fixed memory size\.

###### Definition 2\.6\(VSA Binding\)\.

The binding operator is a map⊗:𝒱×𝒱→𝒱\\otimes:\\mathcal\{V\}\\times\\mathcal\{V\}\\to\\mathcal\{V\}thatpreservesdimensionality\. For a fillerf∈ℝdf\\in\\mathbb\{R\}^\{d\}and a roler∈ℝdr\\in\\mathbb\{R\}^\{d\}, the bound objectTTremains a vector inℝd\\mathbb\{R\}^\{d\}:

T=f⊗r∈ℝdT=f\\otimes r\\in\\mathbb\{R\}^\{d\}\(4\)

###### Definition 2\.7\(VSA Unbinding\)\.

Unbinding⊘:𝒱×𝒱→𝒱\\oslash:\\mathcal\{V\}\\times\\mathcal\{V\}\\to\\mathcal\{V\}also preserves dimensionality\. Given a bound objectT=f⊗rT=f\\otimes r, unbinding with the role gives a version of the filler:

f^=T⊘r\\hat\{f\}=T\\oslash r\(5\)In many architectures, this operation is exact for asinglepair \(f^=f\\hat\{f\}=f\)\. But when unbinding with a superposition of multiple pairs, unbinding givesffplus a pseudo\-random noise term \(from interference of the other stored pairs\)\.

\(f1⊗r1\+f2⊗r2\)⊘r1=f1\+\(f2⊗r2\)⊘r1⏟noise\(f\_\{1\}\\otimes r\_\{1\}\+f\_\{2\}\\otimes r\_\{2\}\)\\oslash r\_\{1\}=f\_\{1\}\+\\underbrace\{\(f\_\{2\}\\otimes r\_\{2\}\)\\oslash r\_\{1\}\}\_\{\\text\{noise\}\}\(6\)

### 2\.3Recognition, Recall, and Clean\-up Memories

Symbolic memory systems can use two distinct query modes\.Recallreconstructs content from a cue: “What is stored at this location?” In contrast,recognitiontests a hypothesis: “Is this specific binding present?” Recognition compares a candidate binding against the memory and returns a similarity score\. Many systems combine unbinding with aclean\-up memory\. This is a codebook of valid symbols\. After unbinding yields a noisy vector, the system can identify the nearest codebook entry\. This effectively converts recall into recognition: testing each vocabulary item and selecting the best match\.

### 2\.4Projection Operators

For a subspaceS⊂ℝdS\\subset\\mathbb\{R\}^\{d\}with orthonormal basis matrixBB, the orthogonal projection ontoSSisPS=BT​BP\_\{S\}=B^\{T\}B\. The complementary projection onto the orthogonal complement isPS⟂=I−BT​BP\_\{S\}^\{\\perp\}=I\-B^\{T\}B\.PS⟂P\_\{S\}^\{\\perp\}is idempotent \(PS⟂​PS⟂=PS⟂P\_\{S\}^\{\\perp\}P\_\{S\}^\{\\perp\}=P\_\{S\}^\{\\perp\}\), andPS​PS⟂=0P\_\{S\}P\_\{S\}^\{\\perp\}=0\(projections are disjoint\)\.

## 3Structural Challenge of Symbolic Memory

Consider the task of storing10001000parse trees, each of depth55, in a symbolic memory\. A standard TPR needs a memory tensor of orderd5d^\{5\}for even modest dimensiond=128d=128\. This needs3×10103\\times 10^\{10\}parameters\. VSAs will keep memory atd=128d=128, which is good\. But superposing10001000bindings drives the SNR to128/1000≈\.358\\sqrt\{128/1000\}\\approx\.358\(close to noise floor\)\.

The challenge\.To support symbolic reasoning, a memory system must accommodate deep recursive structures, allow for superposition of multiple structures within a single trace, and maintain high discrimination capabilities at query time\. These requirements work against one another\.

### 3\.1The Memory Triangle

The tension we describe above manifests as a three\-way trade\-off between structural fidelity, memory footprint, and superposition capacity that we can check quickly\.

Perfect retrieval→\\rightarrowexponential growth\.TPRs place each binding in a distinct tensor dimension\. This gives zero interference\. A tree of depthkkrequiresO​\(dk\)O\(d^\{k\}\)space which is intractable for even modest depth\.

Fixed footprint→\\rightarrowaccumulated interference\.VSAs compress all bindings intoO​\(d\)O\(d\)dimensions\. SuperposingNNitems introduces interference scaling asO​\(N/d\)O\(\\sqrt\{N/d\}\)\. The capacity degrades as memory fills\.

All approaches must occupy some region of the triangle\. Trade\-offs are unavoidable\. The question is aboutwhichtrade\-offs yield favorable scaling given the constraints\.

This trade\-off can also be cast geometrically\. IfNNunit\-norm memory traces are embedded in aDD\-dimensional space, the average squared coherenceμ2=1N​\(N−1\)​∑i≠j\|⟨Ti,Tj⟩\|2\\mu^\{2\}=\\frac\{1\}\{N\(N\-1\)\}\\sum\_\{i\\neq j\}\|\\langle T\_\{i\},T\_\{j\}\\rangle\|^\{2\}is bounded below by the Welch bound\(Welch,[1974](https://arxiv.org/html/2606.11391#bib.bib54)\)

μ2≥N−DD​\(N−1\)whenever​N\>D\.\\mu^\{2\}\\;\\geq\\;\\frac\{N\-D\}\{D\(N\-1\)\}\\quad\\text\{whenever \}N\>D\.\(7\)Once the number of stored items exceeds the effective dimension, cross\-talk cannot be driven to zero\. The relevant design question is therefore not whether interference can be eliminated, but how an architecture allocates effective dimension among context complexity, vocabulary size, and superposition capacity\. OSC takes advantage of an effective dimension ofdpd^\{p\}for the memory tensor while keeping per\-filler storage atp⋅dp\\cdot d\.

### 3\.2Recognition and Local structure can suffice

To navigate this trade\-off, we must re\-evaluate what is strictly necessary for symbolic processing\.

Recognition versus reconstruction\.Most symbolic reasoning systems operate over fixed vocabularies: parse trees use a grammar, programs use a type system\. This prior knowledge shifts the retrieval task in an important way\. Rather than reconstructing an arbitrary vector from a noisy trace \(the hard problem\), the system needs only to determine*which symbol from the codebook is most strongly present*\. This recasts the goal from generative reconstruction to discriminative recognition\. Even VSAs and TPRs with algebraic unbinding rely on this principle\. After unbinding, a cleanup memory maps the noisy result to the nearest vocabulary item\. The “exact” unbinding is discarded in favor of vocabulary search\. If recognition is unavoidable, why pay the cost of maintaining exact unbinding?

Only Local precision\.Structural guarantees need not be universal\. If a system can ensure that fillers relevant to aspecific query contextare geometrically distinct, it can discriminate well – even if fillers in different contexts are only statistically independent\. This suggests a design principle: enforce orthogonalitylocallywithin a context, accept quasi\-orthogonalitygloballyacross contexts\. When querying “subject of this sentence”, it is fine if candidate subjects are distinguishable from each other\. They need not be orthogonal to a verb of an unrelated sentence stored earlier\.

A Geometric Reformulation?These perspectives suggest a shift in how we approach the binding problem\. Rather than expanding the dimension to accommodate all possible structures, we can treat structure as a geometric constraintwithin a fixed spacevia an operation that carves out projected regions for specific roles\. Next, we introduce a special memory architecture that realizes these principles\.

Rejection onto𝒞1\\mathcal\{C\}\_\{1\}𝐏𝒞1⊥=I−𝐫1​𝐫1T\\mathbf\{P\}\_\{\\mathcal\{C\}\_\{1\}\}^\{\\bot\}=I\-\\mathbf\{r\}\_\{1\}\\mathbf\{r\}\_\{1\}^\{T\}𝐏𝒞1⊥​𝐯=𝐯r​e​j\(1\)\\mathbf\{P\}\_\{\\mathcal\{C\}\_\{1\}\}^\{\\bot\}\\mathbf\{v\}=\{\\color\[rgb\]\{0,0,1\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,1\}\\mathbf\{v\}\_\{rej\}^\{\(1\)\}\}Rejection Plane𝐫1\\mathbf\{r\}\_\{1\}𝐯\\mathbf\{v\}𝐯r​e​j\(1\)\\mathbf\{v\}\_\{rej\}^\{\(1\)\}𝐯∥\(1\)\\mathbf\{v\}\_\{\\parallel\}^\{\(1\)\}Rejection onto𝒞2\\mathcal\{C\}\_\{2\}𝐏𝒞2⊥=I−𝐫2​𝐫2T\\mathbf\{P\}\_\{\\mathcal\{C\}\_\{2\}\}^\{\\bot\}=I\-\\mathbf\{r\}\_\{2\}\\mathbf\{r\}\_\{2\}^\{T\}𝐏𝒞2⊥​𝐯=𝐯r​e​j\(2\)\\mathbf\{P\}\_\{\\mathcal\{C\}\_\{2\}\}^\{\\bot\}\\mathbf\{v\}=\{\\color\[rgb\]\{0,0\.5,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.5,0\}\\mathbf\{v\}\_\{rej\}^\{\(2\)\}\}Rejection Plane𝐫2\\mathbf\{r\}\_\{2\}𝐯\\mathbf\{v\}𝐯r​e​j\(2\)\\mathbf\{v\}\_\{rej\}^\{\(2\)\}𝐯∥\(2\)\\mathbf\{v\}\_\{\\parallel\}^\{\(2\)\}Inner Product⟨\\Bigg\\langle,⟩≈0\\Bigg\\rangle\\approx 0

Figure 2:Visualization of the rejection process\. The vector𝐯\\mathbf\{v\}is projected onto orthogonal complements of𝒞1\\mathcal\{C\}\_\{1\}and𝒞2\\mathcal\{C\}\_\{2\}, resulting in nearly orthogonal rejection vectors\. The more vectors in a filler, the less likelihood that all rejection vectors will align\.

## 4Orthogonal Subspace Carving \(OSC\)

Rather than assigning bindings to explicit coordinate slots, our approach will work by defining where information is*not*allowed to go\. Each context is associated with a geometric subspace that acts as a “forbidden” region\. Bindings are then placed in the orthogonal complement, this is the space that remains after carving out the context\.

High level Principle\.The reversal described above is key\. Standard approaches mark storage locations explicitly: “role 1 goes in dimension 7”\. OSC instead says: “this context occupies dimensions\{7,42\}\\\{7,42\\\}\. So, all bindings must avoid them\.” The binding lives in the remainingd−2d\-2dimensions, orthogonal to the context by design\. This exclusion ensures that bindings under different contexts occupydisjoint subspaceswithout requiring coordination, and separation is automatic\. Structure emerges from what is carved away\.

### 4\.1Representational Ingredients\.

The architecture operates within a fixed\-dimensional ambient spaceℝd\\mathbb\{R\}^\{d\}\. Unlike conventional symbolic architectures \(fillers as single vectors\), we construct fillers asstructured objectswith multiple components\.

###### Definition 4\.1\(OSC Filler\)\.

Letppbe a positive integer denoting the order\. The filler spaceℱ\\mathcal\{F\}is defined as the Cartesian product ofppvector spaces \(each of dimensiondd\):

ℱ=\(ℝd\)p\\mathcal\{F\}=\(\\mathbb\{R\}^\{d\}\)^\{p\}\(8\)A specific filler𝐟∈ℱ\\mathbf\{f\}\\in\\mathcal\{F\}is an ordered tuple ofppvectors:

𝐟=\(v1,v2,…,vp\)\\mathbf\{f\}=\(v\_\{1\},v\_\{2\},\\ldots,v\_\{p\}\)\(9\)where each component vectorvi∈ℝdv\_\{i\}\\in\\mathbb\{R\}^\{d\}fori∈\{1,…,p\}i\\in\\\{1,\\ldots,p\\\}\.

Contexts depart from conventional one\-to\-one role\-vector mappings\. Instead of associating each role with a single vector, we associate each context with a subspace that can accommodate multiple roles simultaneously\.

###### Definition 4\.2\(Context\)\.

A context𝒞\\mathcal\{C\}is associated with a subspaceS𝒞⊂ℝdS\_\{\\mathcal\{C\}\}\\subset\\mathbb\{R\}^\{d\}spanned by a set ofkkmutually orthogonal directions\{u1,u2,…,uk\}\\\{u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\\\}\.

Orthogonality is required only among directions within a single context\. Different contexts need not share any geometric relationship\. This realizes the local versus global principle from §[3](https://arxiv.org/html/2606.11391#S3): structural precision within each context but independence across contexts\.

### 4\.2The Carving Operator

To enforce the exclusion principle, we define a projection that removes any overlap with the context subspace\.

###### Definition 4\.3\(Subspace Carving Operation\)\.

Let𝒞\\mathcal\{C\}be a context with basis vectors\{u1,…,uk\}\\\{u\_\{1\},\\ldots,u\_\{k\}\\\}\. We stack these as rows to form the basis matrixB𝒞∈ℝk×dB\_\{\\mathcal\{C\}\}\\in\\mathbb\{R\}^\{k\\times d\}\. The carving operatorP𝒞⟂∈ℝd×dP\_\{\\mathcal\{C\}\}^\{\\perp\}\\in\\mathbb\{R\}^\{d\\times d\}projects onto the null space ofB𝒞B\_\{\\mathcal\{C\}\}:

P𝒞⟂=Id−B𝒞T​B𝒞P\_\{\\mathcal\{C\}\}^\{\\perp\}=I\_\{d\}\-B\_\{\\mathcal\{C\}\}^\{T\}B\_\{\\mathcal\{C\}\}\(10\)

This operation removes any component of a vector that overlaps with the context’s directions\. For any vectorv∈ℝdv\\in\\mathbb\{R\}^\{d\}, the resultP𝒞⟂​vP\_\{\\mathcal\{C\}\}^\{\\perp\}vis guaranteed to be orthogonal to alluiu\_\{i\}, ensuring the processed vector lies entirely outside the context subspaceS𝒞S\_\{\\mathcal\{C\}\}as seen in Figure[2](https://arxiv.org/html/2606.11391#S3.F2)\. We provide motivation for the rejection in Appendix[C](https://arxiv.org/html/2606.11391#A3)\.

### 4\.3Binding Operation

To bind a filler to a context, we apply the carving operator to each component independently\. Then, we fuse the processed components into a higher\-order structure\.

###### Definition 4\.4\(Context Binding\)\.

For a filler𝐟=\(v1,…,vp\)\\mathbf\{f\}=\(v\_\{1\},\\ldots,v\_\{p\}\)and context𝒞\\mathcal\{C\}, the bound objectTboundT\_\{\\text\{bound\}\}is the order\-pptensor product of projected/normalized components:

Tbound=⨂i=1p\(P𝒞⟂​vi‖P𝒞⟂​vi‖\)T\_\{\\text\{bound\}\}=\\bigotimes\_\{i=1\}^\{p\}\\left\(\\frac\{P\_\{\\mathcal\{C\}\}^\{\\perp\}v\_\{i\}\}\{\\\|P\_\{\\mathcal\{C\}\}^\{\\perp\}v\_\{i\}\\\|\}\\right\)\(11\)

By normalizing, the tensor magnitude reflects alignment rather than scale\. Eachviv\_\{i\}is first projected away from context, then normalized\. Then, it is combined via the tensor product\. We get an order\-pptensorTbound∈\(ℝd\)⊗pT\_\{\\text\{bound\}\}\\in\(\\mathbb\{R\}^\{d\}\)^\{\\otimes p\}that encodes the filler while residing fully inS𝒞S\_\{\\mathcal\{C\}\}’s complement\.

### 4\.4Superposition and Memory Formation

The memory𝐌\\mathbf\{M\}is constructed by the superposition of such bound tensors using standard addition \(like TPRs, VSAs\):

𝐌=∑j𝐓bound\(j\)\\mathbf\{M\}=\\sum\_\{j\}\\mathbf\{T\}\_\{\\text\{bound\}\}^\{\(j\)\}\(12\)The enforcement of separation takes place entirely during the binding stage, not at storage\. Because eachTbound\(j\)T\_\{\\text\{bound\}\}^\{\(j\)\}is orthogonal to its respective context subspace and contexts define disjoint subspaces, the bindings naturally separate in the memory space\.This makes memory formation order\-independent \(similar to existing architectures\)\. Arbitrary number of bindings coexist using simple superposition\.

### 4\.5Querying is Recognition

In OSC, we support recognition, not recall\. To query the memory, we construct a candidate binding and measure its similarity against the stored memory\.

###### Definition 4\.5\(Recognition Score\)\.

To test whether a candidate filler𝐠=\(v1,…,vp\)\\mathbf\{g\}=\(v\_\{1\},\\ldots,v\_\{p\}\)is associated with context𝒞\\mathcal\{C\}, construct the candidate tensorTcandT\_\{\\text\{cand\}\}using the same binding mechanism:

Tcand=⨂i=1p\(P𝒞⟂​vi‖P𝒞⟂​vi‖\)T\_\{\\text\{cand\}\}=\\bigotimes\_\{i=1\}^\{p\}\\left\(\\frac\{P\_\{\\mathcal\{C\}\}^\{\\perp\}v\_\{i\}\}\{\\\|P\_\{\\mathcal\{C\}\}^\{\\perp\}v\_\{i\}\\\|\}\\right\)\(13\)The recognition score is the Frobenius inner product:

Score​\(𝐠,𝒞\)=⟨𝐌,Tcand⟩F\\text\{Score\}\(\\mathbf\{g\},\\mathcal\{C\}\)=\\langle\\mathbf\{M\},T\_\{\\text\{cand\}\}\\rangle\_\{F\}\(14\)

When the candidate matches a stored binding, the score is close to 1\. When it does not match, the score is close to zero\. The score yields a unit response for stored items and a zero\-mean response for interference, with a variance that is suppressed as the tensor\-order or dimension increases\.

###### Theorem 4\.6\(Interference Scaling\)\.

Let𝐌\\mathbf\{M\}be a memory superposition ofNNbindings constructed from independent, isotropic random fillers\. For a query matching a stored target, the retrieval scoreSSsatisfies𝔼​\[S\]=1\\mathbb\{E\}\[S\]=1\. The interference noiseIIarising from theN−1N\-1other bindings has zero mean,𝔼​\[I\]=0\\mathbb\{E\}\[I\]=0, and a standard deviation that scales as:

Std​\(S\)=Var​\(Signal\)⏟0\+Var​\(I\)=N−1dp\\text\{Std\}\(S\)=\\sqrt\{\\underbrace\{\\text\{Var\}\(\\text\{Signal\}\)\}\_\{0\}\+\\text\{Var\}\(I\)\}=\\sqrt\{\\frac\{N\-1\}\{d^\{p\}\}\}\(15\)

###### Proof\.

See Appendix[B](https://arxiv.org/html/2606.11391#A2)for the full derivation\. ∎

Retrieval via vocabulary search\.To retrieve the filler associated with a context, we evaluate all candidates from the vocabulary\. For each filler in the codebook, we can construct its binding with the query context and calculate the score\. Then, we select the candidate with the highest score\. This trades algebraic unbinding for geometric interference control, and is favorable in high\-superposition regimes\.

### 4\.6Loss of Exact Unbinding

A structural consequence of this design is that exact algebraic unbinding is impossible\. Because𝐏𝒞⟂\\mathbf\{P\}\_\{\\mathcal\{C\}\}^\{\\perp\}is a projection matrix, it is singular\. Information in the subspace spanned by𝒞\\mathcal\{C\}is discarded\. So, the memory allows queries of the form“Is X stored here?”but not“What is stored here?”without iterating through the vocabulary\. However, as discussed, even architectures with algebraic unbinding \(VSAs, TPRs\) rely on cleanup memories that perform vocabulary search\. Since vocabulary search is unavoidable, the inability to unbind exactly is not a functional disadvantage\. In fact, for VSAs where the unbinding operation is the adjoint of binding \(HRR, MAP, HLB\) retrieval via recognition yields mathematically identical rankings to standard unbinding\-based retrieval, since the adjoint property guarantees the argmax over the codebook is equal \(Appendix[D\.2](https://arxiv.org/html/2606.11391#A4.SS2)\)\.

We call the formulationOrthogonal Subspace Carving\(OSC\)\. It enforces structural distinctions through geometric exclusion: contexts carve forbidden subspaces\. This shifts the goal from reconstruction to discriminative recognition\.

### 4\.7Connection to Clifford algebra

TPR and OSC connect naturally to Clifford algebra, which has attracted growing interest in machine learning, e\.g\., rotor\-based linear layers\(Penceet al\.,[2025](https://arxiv.org/html/2606.11391#bib.bib55)\), general Clifford network layers\(Ruheet al\.,[2023](https://arxiv.org/html/2606.11391#bib.bib57)\), and characterizations of network weights\(Pilanci,[2024](https://arxiv.org/html/2606.11391#bib.bib56)\)\. We show TPR admits a strict generalization within Clifford algebra\. In the Clifford construction, instead of requiring roles and fillers to be supported on disjoint basis vectors, which recovers TPR, they need only be orthogonal to each other for exact recovery of fillers, reducing the tensor memory fromd\|C\|\+1d^\{\\lvert C\\rvert\+1\}to\(d\|C\|\+1\)\\binom\{d\}\{\\lvert C\\rvert\+1\}\.

OSC admits a natural description in this same framework\. The context is encoded as a single algebraic object representing the subspace itself, rather than as a matrix describing it\. The carving operatorP𝒞⟂P\_\{\\mathcal\{C\}\}^\{\\perp\}then becomes a geometric rejection that strips away whatever part of a vector lies inside this subspace, leaving only the component orthogonal to it\. Fillers are likewise lifted from tuples of vectors to subspaces in their own right, and binding combines a filler subspace with the orthogonal complement of the context\. The same orthogonality between fillers and roles that makes the TPR generalization exact is the geometric relationship that OSC enforces through subspace carving\. See Appendix[C](https://arxiv.org/html/2606.11391#A3)for the TPR Clifford algebra generalization and the OSC Clifford equivalent\.

## 5Scalability

We introduced OSC’s binding mechanism, which relies on contexts defining forbidden subspaces\. In this section, we talk about the scalability of roles and fillers\.

### 5\.1Procedural Context Generation\.

Contexts no longer need a global lookup table and can instead depend solely on the abstract roles used to define them, i\.e\., they can be generated procedurally on demand\. The mechanism is simple: combine the symbolic labels of all constituent roles \(e\.g\., “subject”, “sentence\_5”, “parse\_tree”\) into a unique string, hash this string to produce a random seed, and use the seed to generate the context basis \(mutually orthogonal basis vectors𝐁𝒞\\mathbf\{B\}\_\{\\mathcal\{C\}\}\)\. More precisely, for a context𝒞\\mathcal\{C\}defined by role labels\{ℓ1,…,ℓk\}\\\{\\ell\_\{1\},\\ldots,\\ell\_\{k\}\\\}:

1. Step 1Concatenate labels:s=concat​\(ℓ1,…,ℓk\)s=\\text\{concat\}\(\\ell\_\{1\},\\ldots,\\ell\_\{k\}\)
2. Step 2Hash to seed:seed=hash​\(s\)\\text\{seed\}=\\text\{hash\}\(s\)\(e\.g\., SHA\-256\)
3. Step 3Generate random matrix:G∈ℝk×dG\\in\\mathbb\{R\}^\{k\\times d\}using seed
4. Step 4Orthonormalize:B𝒞=QR​\(G\)B\_\{\\mathcal\{C\}\}=\\text\{QR\}\(G\)

Context independence of Memory Footprint\.Because this process is deterministic, the exact orthogonal subspace can be recreated whenever the context is defined\. There is no need to store the basis vectors themselves, decoupling the number of representable contexts from the memory constraints\. Increasing complexity \(associating more roles with a filler\) costs nothing in terms of memory\. This is in sharp contrast to TPRs, where the dimensionality of the representation scales exponentially\.

![Refer to caption](https://arxiv.org/html/2606.11391v1/x1.png)
![Refer to caption](https://arxiv.org/html/2606.11391v1/x2.png)

Figure 3:Performance comparison of1414VSAs atd=4096d=4096and OSC acrossd=25,50,75,100,125,150,175,d=25,50,75,100,125,150,175,and200200forp=2p=2at increasing bundle counts with unique fillers and role depth11\. OSC outperforms all VSAs on both retrieval and recognition tasks\. Note the difference inYYaxes between graphs\. Standard deviation is denoted by the shaded regions\. We provide comparisons for other role depths and VSA dimensions in Appendix[D](https://arxiv.org/html/2606.11391#A4)\.
### 5\.2Filler Representation Efficiency

The multi\-component filler structure provides a second advantage: vocabulary cost grows sub\-linearly with memory capacity\.

##### VSA\.

In VSAs, fillers and memory occupy the samedd\-dimensional space\. A filler isddparameters, so the cost is:

NVSA=d\+\|𝒱\|⋅d=d​\(1\+\|𝒱\|\)N\_\{\\text\{VSA\}\}=d\+\|\\mathcal\{V\}\|\\cdot d=d\(1\+\|\\mathcal\{V\}\|\)\(16\)Each filler costs as many parameters as the entire memory\.

##### OSC Advantage\.

OSC decouples filler dimension from memory dimension\. The memory tensor has effective dimensiond1=dpd\_\{1\}=d^\{p\}, but fillers remain in\(ℝd\)p\(\\mathbb\{R\}^\{d\}\)^\{p\}, requiring onlyp⋅d=p⋅d11/pp\\cdot d=p\\cdot d\_\{1\}^\{1/p\}parameters each:

NOSC=d1\+\|𝒱\|⋅p⋅d11/pN\_\{\\text\{OSC\}\}=d\_\{1\}\+\|\\mathcal\{V\}\|\\cdot p\\cdot d\_\{1\}^\{1/p\}\(17\)Vocabulary cost grows as𝒪​\(d11/p\)\\mathcal\{O\}\(d\_\{1\}^\{1/p\}\)which is sub\-linear in memory capacity\. We show the compounding effect this has on total memory in §[6](https://arxiv.org/html/2606.11391#S6)\.

## 6Experiments

Goals\.We evaluate the scalability and efficiency of OSC through the standard synthetic benchmarks and model integration tasks\. The main goals are:\(G\-1\)Juxtapose memory capacity with total storage efficiency, demonstrating that OSC performs better in terms of total relative size, maintaining a drastically smaller footprint as the number of stored bundles increases\.\(G\-2\)Validate that theconstruction is learnableby integrating OSC into models for common VSA applications\.

Scope of Baselines\.We exclude TPRs from most of our experiments for structural reasons\. Indeed, the entire class of VSAs exists because TPRs become computationally intractable for high context sizes\. In contrast, the performance of OSC is invariant to context complexity but sensitive to the degree of superpositon\. We compare OSC to TPRs where TPRs remain feasible at a role depth of11\.

We evaluate OSC against1414different VSAs, most examined in two recent survey papers\(Schlegelet al\.,[2021](https://arxiv.org/html/2606.11391#bib.bib14); Kleykoet al\.,[2022](https://arxiv.org/html/2606.11391#bib.bib15)\)\. The classical baselines are Holographic Reduced Representations \(HRR\)\(Plate,[1995](https://arxiv.org/html/2606.11391#bib.bib13)\), Fourier\-HRR \(FHRR\)\(Plate,[2003](https://arxiv.org/html/2606.11391#bib.bib17)\), Multiply\-Add\-Permute variants \(MAP\-I, MAP\-C, MAP\-B\)\(Gayler,[1998](https://arxiv.org/html/2606.11391#bib.bib19)\), Binary Spatter Codes \(BSC\)\(Kanerva,[1996](https://arxiv.org/html/2606.11391#bib.bib16)\), and Binary Sparse Distributed Representation variants \(BSDC\-S, BSDC\-CDT\)\(Rachkovskij,[2002](https://arxiv.org/html/2606.11391#bib.bib23)\)\. The more modern baselines are: BSDC\-SEG\(Laihoet al\.,[2015](https://arxiv.org/html/2606.11391#bib.bib24)\), Matrix Binding of Additive Terms \(MBAT\)\(Gallant and Okaywe,[2015](https://arxiv.org/html/2606.11391#bib.bib18)\), Vector\-Derived Transformation Binding \(VTB\)\(Gosmann and Eliasmith,[2019](https://arxiv.org/html/2606.11391#bib.bib12)\), Cyclic Group Representation \(CGR\)\(Yuet al\.,[2022](https://arxiv.org/html/2606.11391#bib.bib21)\), Hadamard\-derived Linear Binding \(HLB\)\(Alamet al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib20)\), and Generalized\-HRR \(GHRR\)\(Yeunget al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib22)\)\.

Table 1:Accompanying table to Figure[3](https://arxiv.org/html/2606.11391#S5.F3)\. Retrieval and recognition accuracy for the best\-performing VSA at22different dimensions and OSC at10001000bundles\. With less memory, OSC outperforms the best VSA on retrieval and recognition tasks\.Why Synthetic Evaluation?Our core scalability experiments rely on synthetic benchmarks, following the standard evaluation for novel VSA architectures\(Schlegelet al\.,[2021](https://arxiv.org/html/2606.11391#bib.bib14); Gosmann and Eliasmith,[2019](https://arxiv.org/html/2606.11391#bib.bib12); Alamet al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib20)\)\. This design choice is grounded in practice: in common VSA use cases, vectors are often fixed randomly generated codebooks \(not learned\)\. Thus, these simulations are not just approximate theoretic limits but reflect expected performance\.

### 6\.1Memory Capacity and Scalability

![Refer to caption](https://arxiv.org/html/2606.11391v1/x3.png)Figure 4:Minimum filler dimension required to achieve99%99\\%accuracy\. A dimension was selected only if mean accuracy across 10 trials exceeded99%99\\%at that dimension but fell below99%99\\%at the next lowest dimension\. The dotted line tracks the actual dimension points, while the solid line represents the best fit\. OSC exhibits the same favorable logarithmic scaling with codebook size as VSAs\.Experimental Setup\.We generate unique roles for every position in the memory\. In the memory analysis, we exclude role storage costs for VSAs since this varies by application \- from just11or22roles to thousands\. While this underestimates the true VSA footprint, the clear difference between VSAs and OSC remains\. We assess recognition and recall\.

![Refer to caption](https://arxiv.org/html/2606.11391v1/x4.png)

Figure 5:Performance of HLB at varying dimensions against OSC forp=3p=3\. Identically colored runs share the same superposition memory size \(not total memory\)\. VSAs are slightly superior than OSC in memory capacity, but OSC’s sub\-linear scaling requires the VSA to have1178×1178\\timesOSC’s total memory to achieve comparable retrieval accuracy\. Table values correspond to1000010000bundles\. Standard deviations \(shaded regions\) are present but negligible\.Results\.Due to the sub\-linear scaling of the filler vocabulary, OSC achieves orders\-of\-magnitude improvements in storage efficiency over VSAs, particularly in high\-superposition regimes\. While standard VSAs typically operate atd=4096d=4096ord=8192d=8192\(with VTB necessitatingd=8100d=8100\), Figure[3](https://arxiv.org/html/2606.11391#S5.F3)and Table[1](https://arxiv.org/html/2606.11391#S6.T1)demonstrates that OSC withd=200d=200andp=2p=2achieves99\.77%99\.77\\%retrieval accuracy for10001000bundles\. This corresponds to a9\.3×9\.3\\timesstorage reduction compared to the best VSA atd=4096d=4096\(13\.18%13\.18\\%accuracy\) and an18\.4×18\.4\\timesreduction compared to the baseline atd=8100d=8100\(35\.88%35\.88\\%accuracy\)\.

The efficiency gap widens significantly at higher levels of superposition\. We selected HLB for this comparison, as it demonstrated the highest performance among VSAs in Figure[3](https://arxiv.org/html/2606.11391#S5.F3)\. As shown in Figure[5](https://arxiv.org/html/2606.11391#S6.F5), OSC atd=80d=80andp=3p=3maintains99\.85%99\.85\\%accuracy for1000010000bundles\. In contrast, maximizing HLB performance required scaling tod=343000d=343000to achieve97\.19%97\.19\\%accuracy, resulting in a1,178×1,178\\timeslarger parameter footprint\. While VSAs exhibit slightly higher capacity when holding memory size constant, this advantage is theoretical rather than practical\. Even when HLB is reduced tod=15625d=15625, yielding0\.56%0\.56\\%accuracy, it remains53\.7×53\.7\\timeslarger than OSC\. Due to the sub\-linear scaling of the filler vocabulary, OSC’s storage efficiency is orders of magnitude smaller in practice, directly supportingG\-1\.

Table 2:Minimum filler dimension required to achieve 99% accuracy\. AtN=10,000N=10,000, OSC has76⋅3=22876\\cdot 3=228parameters per filler, while TPR uses649649parameters, leading to the OSC’s2\.55×2\.55\\timesless parameter usage\.Influence of Codebook Size\.In previous experiments, the codebook size was fixed to match the number of bindings\. However, retrieval performance is inherently dependent on the total size of the codebook, even if the additional fillers are not currently stored in memory\. This occurs because a larger search space increases the probability that a random filler will, by chance, exhibit higher similarity to the noisy retrieved vector than the true target\. This relationship was originally characterized byPlate \([1995](https://arxiv.org/html/2606.11391#bib.bib13)\)and extended to other VSAs bySchlegelet al\.\([2021](https://arxiv.org/html/2606.11391#bib.bib14)\), who showed that codebook capacity scales exponentially with vector dimension\. In Figure[4](https://arxiv.org/html/2606.11391#S6.F4), we show that OSC shares this favorable scaling property: as the codebook size increases while superposition remains constant, the required filler dimension, that which contributes most to total memory, needs to grow only logarithmically to maintain accuracy\. The number of bindings is fixed atN=10N=10\. We provide further experiments in Appendix[D](https://arxiv.org/html/2606.11391#A4)\.

Comparison to TPRsWe compare OSC against TPRs where they remain feasible at a role depth of11\. Following the set\-up in previous codebook size experiment, we find the minimum dimensions for TPR and OSC to reach 99% accuracy across varying levels of superposition\. Table[2](https://arxiv.org/html/2606.11391#S6.T2)shows that OSC achieves this with fewer parameters after a small amount of superposition, and the compression ratio increasingly favors OSC at scale as the filler dimension begins to dominate parameter count\. Note TPR is only exact when the dimension is large enough for orthogonality\.

Inference Speed\.Each VSA has distinct binding and unbinding mechanisms, and the effort invested in low\-level tuning would be uneven across baselines, which makes a comprehensive speed comparison difficult\. We benchmark against HLB, the best\-performing VSA from Figure[3](https://arxiv.org/html/2606.11391#S5.F3), whose binding and unbinding are elementwise multiplication and division, the simplest amongst all VSAs benchmarked against\.

We report retrieval timings with superposition memory held constant and amortized generation costs excluded\. Because HLB’s elementwise operations are simple, it is faster at small enough dimensions\. However, as the representation scales through higher superposition, HLB’s memory footprint dominates\. At that point, even though OSC’s per\-element operations are more expensive, OSC becomes faster because the bottleneck is memory bandwidth, not compute\. Table[3](https://arxiv.org/html/2606.11391#S6.T3)reports wall\-clock retrieval times and parameter counts across three configurations\. See Appendix[D\.4](https://arxiv.org/html/2606.11391#A4.SS4)for a FLOPs comparison\.

### 6\.2Use for Extreme Multi\-label Learning \(XML\)

We show that OSC, like other VSAs, is learnable in models\. To validate this, we apply OSC to the task of Extreme Multi\-label Classification \(XML\)\.

Problem Definition\.XML is a classification setting where the output space consists of a massive number of classes \(L≥100000L\\geq 100000\), but the number of positive labels for any single input is sparse \(K≈10K\\approx 10\)\. Standard neural network approaches \(e\.g\., a final linear layer of sized×Ld\\times L\) scale poorly due to the size of the output space\.

Neuro\-symbolic VSA Approach\.Ganesanet al\.\([2021](https://arxiv.org/html/2606.11391#bib.bib25)\)proposed a neuro\-symbolic approach to solve XML by replacing the computationally expensive final layer with VSA operations\. Instead of learning a classifier for every class, each classkkis assigned a fixed, random atomic vectorckc\_\{k\}\. The model is trained to regress a single superposition vectorSSthat represents the set of all active labels for the input\.

Table 3:Wall\-clock retrieval time and parameter count for HLB vs\. OSC across three benchmarks\.B1: OSC \(d=64,p=2d=64,p=2\) vs HLB \(d=4,096d=4\{,\}096\)\.B2: OSC \(d=200,p=2d=200,p=2\) vs HLB \(d=40,000d=40\{,\}000\)\.B3: OSC \(d=75,p=3d=75,p=3\) vs HLB \(d=421,875d=421\{,\}875\)\.NNdenotes the number of superposed bindings\. A ratio\>1\>1indicates OSC is faster or uses less memory\.Efficient Training via Linearity\.A naive superposition of allLLclasses would remain expensive\. However, by defining two special role vectors—pp\(“present”\) andmm\(“missing”\), we can exploit the linearity of VSAs to shift the compute complexity from the total number of classesO​\(L\)O\(L\)to the number of active classesO​\(K\)O\(K\)\.

The model is trained to give a target vectorSSas:

S=p⊗\(∑i∈Yci\)⏟Active Labels\+m⊗\(∑j∉Ycj\)⏟Inactive LabelsS=\\underbrace\{p\\otimes\\left\(\\sum\_\{i\\in Y\}c\_\{i\}\\right\)\}\_\{\\text\{Active Labels\}\}\+\\underbrace\{m\\otimes\\left\(\\sum\_\{j\\notin Y\}c\_\{j\}\\right\)\}\_\{\\text\{Inactive Labels\}\}\(18\)Calculating the second term \(inactive labels\) is normally expensive\. However, because the sum ofallclass vectorsA=∑k=1LckA=\\sum\_\{k=1\}^\{L\}c\_\{k\}is constant and pre\-computable, we can rewrite the target using the complement of the active set:

S=p⊗\(∑i∈Yci\)\+m⊗\(A−∑i∈Yci\)S=p\\otimes\\left\(\\sum\_\{i\\in Y\}c\_\{i\}\\right\)\+m\\otimes\\left\(A\-\\sum\_\{i\\in Y\}c\_\{i\}\\right\)\(19\)This formulation allows the target to be computed inO​\(K\)O\(K\)time\. The network is then trained using a cosine\-similarity loss that encourages the “present” component of the output to align with the active class vectors, and the “missing” component to differ\.

Table 4:Retrieval performance \(nDCG@5\(↑\)\(\\uparrow\)and PSnDCG@5\(↑\)\(\\uparrow\)\) of VTB, MAP, HLB, and OSC across eight datasets\.Boldindicates the best, whileunderliningdenotes the second best\. OSC ranked first in43\.75%43\.75\\%of cases, surpassing VTB \(37\.5%37\.5\\%\) and HLB \(18\.75%18\.75\\%\)\. In terms of consistency, OSC and VTB both placed in the top two68\.75%68\.75\\%of the time, followed by HLB at62\.5%62\.5\\%\. MAP did not rank in the top two\. Architecture and hyperparameters are detailed in Appendix[E](https://arxiv.org/html/2606.11391#A5)\.Results\.Similar to\(Alamet al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib20)\), we check OSC on 8 benchmark XML datasets\(Bhatiaet al\.,[2016](https://arxiv.org/html/2606.11391#bib.bib51)\)using normalized discounted cumulative gain \(nDCG\) and propensity\-scored \(PS\) based nDCG \(PSnDCG\) as suggested by\(Jainet al\.,[2016](https://arxiv.org/html/2606.11391#bib.bib26)\)\. Table[4](https://arxiv.org/html/2606.11391#S6.T4)shows that OSC learns effectively here, yielding performance competitive with MAP, HLB, and VTB\. This confirms that our binding mechanism preserves learnability in gradient\-based systems, directly supportingG\-2\.

## 7Related Works

VSAs and TPRs are extensively covered in §[1](https://arxiv.org/html/2606.11391#S1)and §[2](https://arxiv.org/html/2606.11391#S2)\.

Matrix Memories and Outer Product Storage\.Distributed associative memories based on outer product storage emerged independently in the early 1970s\(Kohonen,[2009](https://arxiv.org/html/2606.11391#bib.bib27); Anderson,[1972](https://arxiv.org/html/2606.11391#bib.bib28); Nakano,[2007](https://arxiv.org/html/2606.11391#bib.bib29)\)\. The pseudo\-inverse extension, Optimal Linear Associative Memory \(OLAM\)\(Kohonen and Ruohonen,[1973](https://arxiv.org/html/2606.11391#bib.bib30)\), replaces the correlation sum with the Moore\-Penrose solution, achieving perfect retrieval up to the dimensionality limit\. Hopfield networks\(Hopfield,[1982](https://arxiv.org/html/2606.11391#bib.bib31)\)popularized this framework by introducing energy\-based dynamics and stability analysis\(McElieceet al\.,[2003](https://arxiv.org/html/2606.11391#bib.bib32); Baldi and Venkatesh,[1987](https://arxiv.org/html/2606.11391#bib.bib33)\)\. Modern work on dense associative memories\(Krotov and Hopfield,[2016](https://arxiv.org/html/2606.11391#bib.bib34); Demircigilet al\.,[2017](https://arxiv.org/html/2606.11391#bib.bib35)\)showed that higher\-order interactions enable exponential storage capacity, connecting classical memory models to deep learning\(Krotov and Hopfield,[2018](https://arxiv.org/html/2606.11391#bib.bib36)\)\. Recent analyses characterize capacity under various conditions\(Lucibello and Mézard,[2024](https://arxiv.org/html/2606.11391#bib.bib37); Huet al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib38)\)and extend these models to kernelized settings\(Wuet al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib39)\)\.

Conceptors and Gradient Projection Methods\.Conceptors\(Jaeger,[2014](https://arxiv.org/html/2606.11391#bib.bib40),[2017](https://arxiv.org/html/2606.11391#bib.bib41)\)provide soft projection matrices for managing recurrent network dynamics, with Boolean operations enabling compositional control\. Conceptor\-Aided Backpropagation\(He and Jaeger,[2018](https://arxiv.org/html/2606.11391#bib.bib42)\)applies this to continual learning by projecting weight updates onto null spaces of previous tasks\. Orthogonal gradient projection methods pursue the same goal through different mechanisms: OWM\(Zenget al\.,[2019](https://arxiv.org/html/2606.11391#bib.bib43)\)maintains projectors from input representations, OGD\(Farajtabaret al\.,[2020](https://arxiv.org/html/2606.11391#bib.bib44)\)stores gradient directions directly, and GPM\(Sahaet al\.,[2021](https://arxiv.org/html/2606.11391#bib.bib45)\)uses SVD on activations to find core gradient subspaces\. Extensions include scaled gradient projection\(Saha and Roy,[2023](https://arxiv.org/html/2606.11391#bib.bib46)\), class\-level gradients\(Chenet al\.,[2022](https://arxiv.org/html/2606.11391#bib.bib47)\), and adaptive orthogonal projection\(Guoet al\.,[2022](https://arxiv.org/html/2606.11391#bib.bib48)\)\. These ideas have been applied to LLM fine\-tuning via O\-LoRA\(Wanget al\.,[2023](https://arxiv.org/html/2606.11391#bib.bib49)\)and related methods\(Tanget al\.,[2026](https://arxiv.org/html/2606.11391#bib.bib50)\)\.

## 8Conclusion

We introduce Orthogonal Subspace Carving, a tensor memory architecture that projects fillers onto role null spaces before superposition, geometrically suppressing cross\-talk between bound structures\. OSC decouples context complexity from performance via procedural basis generation and offers significantly reduced memory footprint compared to traditional VSAs and TPRs due to the sub\-linear scaling of the filler dimension\. Finally, we demonstrate the compatibility of OSC with gradient\-based models by validating its performance on XML datasets\. See Appendix[A](https://arxiv.org/html/2606.11391#A1)for limitations and future work\.

## Acknowledgments

We thank the anonymous reviewers for their constructive feedback\. Authors were all partly supported by NIH R01AG092220\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- M\. M\. Alam, A\. Oberle, E\. Raff, S\. Biderman, T\. Oates, and J\. Holt \(2024\)A walsh hadamard derived linear vector symbolic architecture\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 2711–2733\.External Links:[Document](https://dx.doi.org/10.52202/079017-0089),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/0525fa17a8dbea687359116d01732e12-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2606.11391#A1.p2.1),[Table 6](https://arxiv.org/html/2606.11391#A5.T6),[Table 6](https://arxiv.org/html/2606.11391#A5.T6.2.1),[§6\.2](https://arxiv.org/html/2606.11391#S6.SS2.p6.1),[§6](https://arxiv.org/html/2606.11391#S6.p3.1),[§6](https://arxiv.org/html/2606.11391#S6.p4.1)\.
- J\. A\. Anderson \(1972\)A simple neural network generating an interactive memory\.Mathematical Biosciences14\(3\),pp\. 197–220\.External Links:ISSN 0025\-5564,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0025-5564%2872%2990075-2),[Link](https://www.sciencedirect.com/science/article/pii/0025556472900752)Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- P\. Baldi and S\. S\. Venkatesh \(1987\)Number of stable points for spin\-glasses and neural networks of higher orders\.Physical Review Letters58\(9\),pp\. 913\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- K\. Bhatia, K\. Dahiya, H\. Jain, P\. Kar, A\. Mittal, Y\. Prabhu, and M\. Varma \(2016\)The extreme classification repository: multi\-label datasets and code\.External Links:[Link](http://manikvarma.org/downloads/XC/XMLRepository.html)Cited by:[§6\.2](https://arxiv.org/html/2606.11391#S6.SS2.p6.1)\.
- C\. Chen, J\. Zhang, J\. Song, and L\. Gao \(2022\)Class gradient projection for continual learning\.InProceedings of the 30th ACM International Conference on Multimedia,pp\. 5575–5583\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- E\. Chisolm \(2012\)Geometric algebra\.External Links:1205\.5935,[Link](https://arxiv.org/abs/1205.5935)Cited by:[Appendix C](https://arxiv.org/html/2606.11391#A3.p1.1)\.
- M\. Demircigil, J\. Heusel, M\. Löwe, S\. Upgang, and F\. Vermet \(2017\)On a model of associative memory with huge storage capacity\.Journal of Statistical Physics168\(2\),pp\. 288–299\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- C\. Doran and A\. Lasenby \(2003\)Geometric algebra for physicists\.Cambridge University Press\.Cited by:[Lemma C\.3](https://arxiv.org/html/2606.11391#A3.Thmtheorem3.1.p1.1.1)\.
- N\. Dziri, X\. Lu, M\. Sclar, X\. L\. Li, L\. Jiang, B\. Y\. Lin, P\. West, C\. Bhagavatula, R\. L\. Bras, J\. D\. Hwang, S\. Sanyal, S\. Welleck, X\. Ren, A\. Ettinger, Z\. Harchaoui, and Y\. Choi \(2023\)Faith and fate: limits of transformers on compositionality\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- M\. Farajtabar, N\. Azizan, A\. Mott, and A\. Li \(2020\)Orthogonal gradient descent for continual learning\.InInternational conference on artificial intelligence and statistics,pp\. 3762–3773\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- S\. I\. Gallant and T\. W\. Okaywe \(2015\)Representing objects, relations, and sequences\.External Links:1501\.07627,[Link](https://arxiv.org/abs/1501.07627)Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- A\. Ganesan, H\. Gao, S\. Gandhi, E\. Raff, T\. Oates, J\. Holt, and M\. McLean \(2021\)Learning with holographic reduced representations\.Advances in neural information processing systems34,pp\. 25606–25620\.Cited by:[§6\.2](https://arxiv.org/html/2606.11391#S6.SS2.p3.3)\.
- R\. W\. Gayler \(1998\)Multiplicative binding, representation operators & analogy\.External Links:[Link](https://api.semanticscholar.org/CorpusID:14487970)Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- J\. Gosmann and C\. Eliasmith \(2019\)Vector\-derived transformation binding: an improved binding operation for deep symbol\-like processing in neural networks\.Neural Computation31\(5\),pp\. 849–869\.External Links:ISSN 0899\-7667,[Document](https://dx.doi.org/10.1162/neco%5Fa%5F01179),[Link](https://doi.org/10.1162/neco_a_01179),https://direct\.mit\.edu/neco/article\-pdf/31/5/849/1052783/neco\_a\_01179\.pdfCited by:[§1](https://arxiv.org/html/2606.11391#S1.p3.1),[§6](https://arxiv.org/html/2606.11391#S6.p3.1),[§6](https://arxiv.org/html/2606.11391#S6.p4.1)\.
- Y\. Guo, W\. Hu, D\. Zhao, and B\. Liu \(2022\)Adaptive orthogonal projection for batch and online continual learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 6783–6791\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- X\. He and H\. Jaeger \(2018\)Overcoming catastrophic interference using conceptor\-aided backpropagation\.InInternational Conference on Learning Representations,Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- J\. J\. Hopfield \(1982\)Neural networks and physical systems with emergent collective computational abilities\.\.Proceedings of the national academy of sciences79\(8\),pp\. 2554–2558\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- J\. Y\. Hu, D\. Wu, and H\. Liu \(2024\)Provably optimal memory capacity for modern hopfield models: transformer\-compatible dense associative memories as spherical codes\.Advances in Neural Information Processing Systems37,pp\. 70693–70729\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- H\. Jaeger \(2014\)Controlling recurrent neural networks by conceptors\.arXiv preprint arXiv:1403\.3369\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- H\. Jaeger \(2017\)Using conceptors to manage neural long\-term memories for temporal patterns\.Journal of Machine Learning Research18\(13\),pp\. 1–43\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- H\. Jain, Y\. Prabhu, and M\. Varma \(2016\)Extreme multi\-label loss functions for recommendation, tagging, ranking & other missing label applications\.InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining,pp\. 935–944\.Cited by:[§6\.2](https://arxiv.org/html/2606.11391#S6.SS2.p6.1)\.
- P\. Kanerva \(1996\)Binary spatter\-coding of ordered k\-tuples\.InInternational Conference on Artificial Neural Networks,External Links:[Link](https://api.semanticscholar.org/CorpusID:12261165)Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- P\. Kanerva \(2009\)Hyperdimensional computing: an introduction to computing in distributed representation with high\-dimensional random vectors\.Cognitive Computation1\(2\),pp\. 139–159\.Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1),[§1](https://arxiv.org/html/2606.11391#S1.p3.1)\.
- D\. Keysers, N\. Schärli, N\. Scales, H\. Buisman, D\. Furrer, S\. Kashubin, N\. Momchev, D\. Sinopalnikov, L\. Stafiniak, T\. Tihon,et al\.\(2020\)Measuring compositional generalization: a comprehensive method on realistic data\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SygcCnNKwr)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- N\. Kim and T\. Linzen \(2020\)COGS: a compositional generalization challenge based on semantic interpretation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9087–9105\.Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- D\. Kleyko, D\. A\. Rachkovskij, E\. Osipov, and A\. Rahimi \(2022\)A survey on hyperdimensional computing aka vector symbolic architectures, part i: models and data transformations\.ACM Computing Surveys55\(6\),pp\. 1–40\.External Links:ISSN 1557\-7341,[Link](http://dx.doi.org/10.1145/3538531),[Document](https://dx.doi.org/10.1145/3538531)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1),[§1](https://arxiv.org/html/2606.11391#S1.p3.1),[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- T\. Kohonen and M\. Ruohonen \(1973\)Representation of associated data by matrix operators\.IEEE Transactions on Computers100\(7\),pp\. 701–702\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- T\. Kohonen \(2009\)Correlation matrix memories\.IEEE transactions on computers100\(4\),pp\. 353–359\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- D\. Krotov and J\. J\. Hopfield \(2016\)Dense associative memory for pattern recognition\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- D\. Krotov and J\. Hopfield \(2018\)Dense associative memory is robust to adversarial inputs\.Neural computation30\(12\),pp\. 3151–3167\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- M\. Laiho, J\. H\. Poikonen, P\. Kanerva, and E\. Lehtonen \(2015\)High\-dimensional computing with sparse vectors\.In2015 IEEE Biomedical Circuits and Systems Conference \(BioCAS\),pp\. 1–4\.Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- B\. Li, L\. Donatelli, A\. Koller, T\. Linzen, Y\. Yao, and N\. Kim \(2023\)SLOG: a structural generalization benchmark for semantic parsing\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 3213–3232\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.194/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.194)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- C\. Lucibello and M\. Mézard \(2024\)Exponential capacity of dense associative memories\.Physical Review Letters132\(7\),pp\. 077301\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- G\. F\. Marcus \(2001\)The algebraic mind: integrating connectionism and cognitive science\.MIT press\.Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- R\. McEliece, E\. Posner, E\. Rodemich, and S\. Venkatesh \(2003\)The capacity of the hopfield associative memory\.IEEE transactions on Information Theory33\(4\),pp\. 461–482\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- K\. Nakano \(2007\)Associatron\-a model of associative memory\.IEEE Transactions on Systems, Man, and Cybernetics\(3\),pp\. 380–388\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- A\. Newell \(1980\)Physical symbol systems\.Cognitive Science4\(2\),pp\. 135–183\.Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- A\. Newell \(1982\)The knowledge level\.Artificial Intelligence18\(1\),pp\. 87–127\.Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p1.1)\.
- T\. Pence, D\. Yamada, and V\. Singh \(2025\)Composing linear layers from irreducibles\.InAdvances in Neural Information Processing Systems,Cited by:[§4\.7](https://arxiv.org/html/2606.11391#S4.SS7.p1.2)\.
- M\. Pilanci \(2024\)From complexity to clarity: analytical expressions of deep neural network weights via clifford algebra and convexity\.Transactions on machine learning research\.Cited by:[§4\.7](https://arxiv.org/html/2606.11391#S4.SS7.p1.2)\.
- T\. A\. Plate \(1995\)Holographic reduced representations\.IEEE Transactions on Neural Networks6\(3\),pp\. 623–641\.External Links:[Document](https://dx.doi.org/10.1109/72.377968)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p3.1),[§6\.1](https://arxiv.org/html/2606.11391#S6.SS1.p4.1),[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- T\. A\. Plate \(2003\)Holographic reduced representation: distributed representation for cognitive structures\.CSLI Publications\.Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- D\. A\. Rachkovskij \(2002\)Representation and processing of structures with binary sparse distributed codes\.IEEE transactions on Knowledge and Data Engineering13\(2\),pp\. 261–276\.Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- D\. Ruhe, J\. K\. Gupta, S\. de Keninck, M\. Welling, and J\. Brandstetter \(2023\)Geometric clifford algebra networks\.International Conference on Machine Learning\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2302.06594)Cited by:[§4\.7](https://arxiv.org/html/2606.11391#S4.SS7.p1.2)\.
- G\. Saha, I\. Garg, and K\. Roy \(2021\)Gradient projection memory for continual learning\.arXiv preprint arXiv:2103\.09762\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- G\. Saha and K\. Roy \(2023\)Continual learning with scaled gradient projection\.InProceedings of the AAAI conference on artificial intelligence,Vol\.37,pp\. 9677–9685\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- K\. Schlegel, P\. Neubert, and P\. Protzel \(2021\)A comparison of vector symbolic architectures\.Artificial Intelligence Review55\(6\),pp\. 4523–4555\.External Links:ISSN 1573\-7462,[Link](http://dx.doi.org/10.1007/s10462-021-10110-3),[Document](https://dx.doi.org/10.1007/s10462-021-10110-3)Cited by:[§6\.1](https://arxiv.org/html/2606.11391#S6.SS1.p4.1),[§6](https://arxiv.org/html/2606.11391#S6.p3.1),[§6](https://arxiv.org/html/2606.11391#S6.p4.1)\.
- P\. Smolensky \(1990\)Tensor product variable binding and the representation of symbolic structures in connectionist systems\.Artificial Intelligence46\(1\),pp\. 159–216\.External Links:ISSN 0004\-3702,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0004-3702%2890%2990007-M),[Link](https://www.sciencedirect.com/science/article/pii/000437029090007M)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p2.2)\.
- P\. Soulos, H\. Conklin, M\. Opper, P\. Smolensky, J\. Gao, and R\. Fernandez \(2024\)Compositional generalization across distributional shifts with sparse tree operations\.External Links:2412\.14076,[Link](https://arxiv.org/abs/2412.14076)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p2.2)\.
- P\. Soulos, E\. Hu, K\. McCurdy, Y\. Chen, R\. Fernandez, P\. Smolensky, and J\. Gao \(2023\)Differentiable tree operations promote compositional generalization\.External Links:2306\.00751,[Link](https://arxiv.org/abs/2306.00751)Cited by:[§1](https://arxiv.org/html/2606.11391#S1.p2.2)\.
- P\. Tang, X\. Hu, Y\. Liu, L\. Ding, D\. Zhang, X\. Wu, and D\. Zhang \(2026\)Put the space of lora initialization to the extreme to preserve pre\-trained knowledge\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33232–33240\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. Huang \(2023\)Orthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10658–10671\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.
- L\. Welch \(1974\)Lower bounds on the maximum cross correlation of signals \(corresp\.\)\.IEEE Transactions on Information Theory20\(3\),pp\. 397–399\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1974.1055219)Cited by:[§3\.1](https://arxiv.org/html/2606.11391#S3.SS1.p5.3)\.
- D\. Wu, J\. Y\. Hu, T\. Hsiao, and H\. Liu \(2024\)Uniform memory retrieval with larger capacity for modern hopfield models\.arXiv preprint arXiv:2404\.03827\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p2.1)\.
- C\. Yeung, Z\. Zou, and M\. Imani \(2024\)Generalized holographic reduced representations\.External Links:2405\.09689,[Link](https://arxiv.org/abs/2405.09689)Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- T\. Yu, Y\. Zhang, Z\. Zhang, and C\. M\. De Sa \(2022\)Understanding hyperdimensional computing for parallel single\-pass learning\.Advances in neural information processing systems35,pp\. 1157–1169\.Cited by:[§6](https://arxiv.org/html/2606.11391#S6.p3.1)\.
- G\. Zeng, Y\. Chen, B\. Cui, and S\. Yu \(2019\)Continual learning of context\-dependent processing in neural networks\.Nature Machine Intelligence1\(8\),pp\. 364–372\.Cited by:[§7](https://arxiv.org/html/2606.11391#S7.p3.1)\.

## Appendix ALimitations and Future Work

As shown in Figure[8](https://arxiv.org/html/2606.11391#A4.F8)and[9](https://arxiv.org/html/2606.11391#A4.F9), the representation density of OSC is slightly lower than that of traditional VSAs\. While OSC significantly outperforms VSAs in terms of total memory, VSAs are slightly better when superposition memory, not total storage, is the constraint\.

As illustrated in Figure[1](https://arxiv.org/html/2606.11391#S2.F1), in VSAs the bind operation produces a vector dissimilar to its constituents, while the bundle operation preserves similarity\. OSC similarly preserves similarity during bundling, however unlike VSAs, it allows the binding operation to do so as well\. In the extreme case, if a filler vector lies entirely within the orthogonal complement of the context, the rejection operation leaves the filler unchanged\. Given that VSAs have established applications in information hiding\(Alamet al\.,[2024](https://arxiv.org/html/2606.11391#bib.bib20)\), exploring the implications of this preserved similarity and mechanisms for encouraging dissimilarity in OSC remains a promising direction for future work\.

## Appendix BRetrieval Statistics

In this appendix, we derive the mean and variance of the retrieval scoreSSfrom Theorem[4\.6](https://arxiv.org/html/2606.11391#S4.Thmtheorem6)\. We assume that the component vectorsv∈ℝdv\\in\\mathbb\{R\}^\{d\}are independent random vectors drawn from an isotropic Gaussian distribution, and that each binding uses an independently generated context\.

### B\.1Score Definition

The retrieval scoreSSis defined as the Frobenius inner product between the memory tensorM=∑j=1NT\(j\)M=\\sum\_\{j=1\}^\{N\}T^\{\(j\)\}and a candidate query tensorT\(q\)T^\{\(q\)\}\.

Recall that each bound tensorTTis the order\-pptensor product of the projected, normalized filler components\. Letv~i\(j\)\\tilde\{v\}^\{\(j\)\}\_\{i\}denote theii\-th component of thejj\-th filler after projection and normalization:

v~i\(j\)=PCj⟂​vi\(j\)‖PCj⟂​vi\(j\)‖\\tilde\{v\}^\{\(j\)\}\_\{i\}=\\frac\{P^\{\\perp\}\_\{C\_\{j\}\}v^\{\(j\)\}\_\{i\}\}\{\\\|P^\{\\perp\}\_\{C\_\{j\}\}v^\{\(j\)\}\_\{i\}\\\|\}\(20\)
The pairwise inner product between two tensor termsZj​q=⟨T\(j\),T\(q\)⟩FZ\_\{jq\}=\\langle T^\{\(j\)\},T^\{\(q\)\}\\rangle\_\{F\}factors into the product of component inner products:

Zj​q=∏i=1p⟨v~i\(j\),v~i\(q\)⟩Z\_\{jq\}=\\prod\_\{i=1\}^\{p\}\\langle\\tilde\{v\}^\{\(j\)\}\_\{i\},\\tilde\{v\}^\{\(q\)\}\_\{i\}\\rangle\(21\)

### B\.2Inner Product Statistics for Independent Contexts

Before analyzing the moments ofZj​qZ\_\{jq\}, we establish the key statistical property of inner products between vectors projected onto different random subspaces\.

LetP1⟂,P2⟂P\_\{1\}^\{\\perp\},P\_\{2\}^\{\\perp\}be projections onto independently generated random\(d−k\)\(d\-k\)\-dimensional subspaces\. Letu,v∼𝒩​\(0,Id\)u,v\\sim\\mathcal\{N\}\(0,I\_\{d\}\)be independent, and define:

u~=P1⟂​u‖P1⟂​u‖,v~=P2⟂​v‖P2⟂​v‖\\tilde\{u\}=\\frac\{P\_\{1\}^\{\\perp\}u\}\{\\\|P\_\{1\}^\{\\perp\}u\\\|\},\\quad\\tilde\{v\}=\\frac\{P\_\{2\}^\{\\perp\}v\}\{\\\|P\_\{2\}^\{\\perp\}v\\\|\}\(22\)
Claim:𝔼​\[⟨u~,v~⟩2\]=1d\\mathbb\{E\}\[\\langle\\tilde\{u\},\\tilde\{v\}\\rangle^\{2\}\]=\\frac\{1\}\{d\}

###### Proof\.

Conditioned on the subspaces,u~\\tilde\{u\}andv~\\tilde\{v\}are independent uniform unit vectors in their respective\(d−k\)\(d\-k\)\-dimensional subspaces\. LetE,F∈ℝd×\(d−k\)E,F\\in\\mathbb\{R\}^\{d\\times\(d\-k\)\}be orthonormal bases for these subspaces\. Thenu~=E​α\\tilde\{u\}=E\\alphaandv~=F​β\\tilde\{v\}=F\\betawhereα,β\\alpha,\\betaare uniform onSd−k−1S^\{d\-k\-1\}\.

The inner product is:

⟨u~,v~⟩=αT​ET​F​β=αT​G​β\\langle\\tilde\{u\},\\tilde\{v\}\\rangle=\\alpha^\{T\}E^\{T\}F\\beta=\\alpha^\{T\}G\\beta\(23\)whereG=ET​FG=E^\{T\}F\. For uniform vectors onSd−k−1S^\{d\-k\-1\}:

𝔼​\[⟨u~,v~⟩2∣G\]=‖G‖F2\(d−k\)2\\mathbb\{E\}\[\\langle\\tilde\{u\},\\tilde\{v\}\\rangle^\{2\}\\mid G\]=\\frac\{\\\|G\\\|\_\{F\}^\{2\}\}\{\(d\-k\)^\{2\}\}\(24\)
SinceP2⟂=F​FTP\_\{2\}^\{\\perp\}=FF^\{T\}:

‖G‖F2=‖ET​F‖F2=tr​\(ET​F​FT​E\)=tr​\(ET​P2⟂​E\)\\\|G\\\|\_\{F\}^\{2\}=\\\|E^\{T\}F\\\|\_\{F\}^\{2\}=\\mathrm\{tr\}\(E^\{T\}FF^\{T\}E\)=\\mathrm\{tr\}\(E^\{T\}P\_\{2\}^\{\\perp\}E\)\(25\)
Taking expectations over independent random subspaces,𝔼​\[P2⟂\]=d−kd​Id\\mathbb\{E\}\[P\_\{2\}^\{\\perp\}\]=\\frac\{d\-k\}\{d\}I\_\{d\}, so:

𝔼​\[‖G‖F2\]=d−kd​tr​\(ET​E\)=\(d−k\)2d\\mathbb\{E\}\[\\\|G\\\|\_\{F\}^\{2\}\]=\\frac\{d\-k\}\{d\}\\mathrm\{tr\}\(E^\{T\}E\)=\\frac\{\(d\-k\)^\{2\}\}\{d\}\(26\)
Therefore:

𝔼​\[⟨u~,v~⟩2\]=\(d−k\)2/d\(d−k\)2=1d\\mathbb\{E\}\[\\langle\\tilde\{u\},\\tilde\{v\}\\rangle^\{2\}\]=\\frac\{\(d\-k\)^\{2\}/d\}\{\(d\-k\)^\{2\}\}=\\frac\{1\}\{d\}\(27\)∎

### B\.3Moment Analysis

We now analyze the moments ofZj​qZ\_\{jq\}using the result above\.

1\. Signal Term \(j=qj=q\)\.In the case of a match, we are taking the inner product of identical unit vectors\.

⟨v~i\(q\),v~i\(q\)⟩=1⟹Zq​q=∏i=1p\(1\)=1\\langle\\tilde\{v\}^\{\(q\)\}\_\{i\},\\tilde\{v\}^\{\(q\)\}\_\{i\}\\rangle=1\\implies Z\_\{qq\}=\\prod\_\{i=1\}^\{p\}\(1\)=1\(28\)
Thus, the signal is deterministic and so

𝔼​\[Zq​q\]=1,Var​\(Zq​q\)=0\\mathbb\{E\}\[Z\_\{qq\}\]=1,\\quad\\mathrm\{Var\}\(Z\_\{qq\}\)=0\(29\)
2\. Interference Term \(j≠qj\\neq q\)\.In the case of a mismatch,v~i\(j\)\\tilde\{v\}^\{\(j\)\}\_\{i\}andv~i\(q\)\\tilde\{v\}^\{\(q\)\}\_\{i\}are derived from independent Gaussian vectors projected onto independently generated\(d−k\)\(d\-k\)\-dimensional subspaces\. By the claim above:

𝔼​\[⟨v~i\(j\),v~i\(q\)⟩\]=0,𝔼​\[⟨v~i\(j\),v~i\(q\)⟩2\]=1d\\mathbb\{E\}\[\\langle\\tilde\{v\}^\{\(j\)\}\_\{i\},\\tilde\{v\}^\{\(q\)\}\_\{i\}\\rangle\]=0,\\quad\\mathbb\{E\}\[\\langle\\tilde\{v\}^\{\(j\)\}\_\{i\},\\tilde\{v\}^\{\(q\)\}\_\{i\}\\rangle^\{2\}\]=\\frac\{1\}\{d\}\(30\)
Since the componentsi=1​…​pi=1\\ldots pare independent, the expectation of the product is the product of the expectations:

𝔼​\[Zj​q\]=∏i=1p𝔼​\[⟨v~i\(j\),v~i\(q\)⟩\]=0\\mathbb\{E\}\[Z\_\{jq\}\]=\\prod\_\{i=1\}^\{p\}\\mathbb\{E\}\[\\langle\\tilde\{v\}^\{\(j\)\}\_\{i\},\\tilde\{v\}^\{\(q\)\}\_\{i\}\\rangle\]=0\(31\)
The variance calculation follows:

Var​\(Zj​q\)=𝔼​\[Zj​q2\]=∏i=1p𝔼​\[⟨v~i\(j\),v~i\(q\)⟩2\]=\(1d\)p=1dp\\mathrm\{Var\}\(Z\_\{jq\}\)=\\mathbb\{E\}\[Z\_\{jq\}^\{2\}\]=\\prod\_\{i=1\}^\{p\}\\mathbb\{E\}\[\\langle\\tilde\{v\}^\{\(j\)\}\_\{i\},\\tilde\{v\}^\{\(q\)\}\_\{i\}\\rangle^\{2\}\]=\\left\(\\frac\{1\}\{d\}\\right\)^\{p\}=\\frac\{1\}\{d^\{p\}\}\(32\)

### B\.4Total Retrieval Variance

The retrieval score isS=∑j=1NZj​q=Zq​q\+∑j≠qZj​q=1\+IS=\\sum\_\{j=1\}^\{N\}Z\_\{jq\}=Z\_\{qq\}\+\\sum\_\{j\\neq q\}Z\_\{jq\}=1\+I, whereI=∑j≠qZj​qI=\\sum\_\{j\\neq q\}Z\_\{jq\}is the interference\. Since the interference terms are independent with zero mean and variance1dp\\frac\{1\}\{d^\{p\}\}:

𝔼​\[S\]=1,Var​\(S\)=Var​\(I\)=N−1dp\\mathbb\{E\}\[S\]=1,\\quad\\mathrm\{Var\}\(S\)=\\mathrm\{Var\}\(I\)=\\frac\{N\-1\}\{d^\{p\}\}\(33\)

## Appendix CClifford Generalization of TPR

Chisolm \([2012](https://arxiv.org/html/2606.11391#bib.bib53)\)provides the needed background on Clifford algebra\. We conclude this section by providing motivation for the rejection operator\.

### C\.1Motivation

The carving operatorP𝒞⟂P\_\{\\mathcal\{C\}\}^\{\\perp\}from Definition 4\.3 can be expressed in Clifford algebra without matrices\. The idea is simple: instead of representing the context subspace with a basis matrix, we encode it as a single object called a blade\.

Let𝒞\\mathcal\{C\}be a context with orthogonal vectorsr1,…,rkr\_\{1\},\\ldots,r\_\{k\}\. Their wedge productAr=r1∧⋯∧rkA\_\{r\}=r\_\{1\}\\wedge\\cdots\\wedge r\_\{k\}is akk\-blade representing the subspace they span\. This blade*is*the context, not a matrix describing it, but the subspace itself as an algebraic object\.

Clifford algebra provides a natural decomposition of any vectoraainto components parallel and perpendicular to this subspace:

PAr​\(a\)\+RAr​\(a\)=a,wherePAr​\(a\)=\(a⌟Ar\)​Ar−1\.P\_\{A\_\{r\}\}\(a\)\+R\_\{A\_\{r\}\}\(a\)=a,\\quad\\text\{where\}\\quad P\_\{A\_\{r\}\}\(a\)=\(a\\mathbin\{\\lrcorner\}A\_\{r\}\)A\_\{r\}^\{\-1\}\.The projectionPAr​\(a\)P\_\{A\_\{r\}\}\(a\)lies inside the context; the rejectionRAr​\(a\)R\_\{A\_\{r\}\}\(a\)is what remains after removing that component\. This rejection is exactly the carving operation \- it strips away whatever part ofaaoverlaps with the forbidden subspace\.

In this formulation, fillers also become subspaces\. A filler is now app\-bladeF=v1∧⋯∧vpF=v\_\{1\}\\wedge\\cdots\\wedge v\_\{p\}rather than a tuple of vectors\. To bind these two subspaces, we reject each component individually and wedge the results:

RAr​\(v1∧⋯∧vp\):=RAr​\(v1\)∧⋯∧RAr​\(vp\)\.R\_\{A\_\{r\}\}\(v\_\{1\}\\wedge\\cdots\\wedge v\_\{p\}\):=R\_\{A\_\{r\}\}\(v\_\{1\}\)\\wedge\\cdots\\wedge R\_\{A\_\{r\}\}\(v\_\{p\}\)\.The output is a blade lying entirely in the orthogonal complement of the context, which is precisely what OSC binding requires\. This perspective offers geometric clarity, but proving properties in this setting is more involved, so we use the GPU\-friendly linear algebra formulation throughout the paper\.

OSC is not alone in admitting a Clifford\-algebraic formulation\. We generalize TPR using the framework of Clifford algebras\. LetCl​\(n\)\\mathrm\{Cl\}\(n\)denote the Clifford algebra generated bynnbasis vectors squaring to11\. We show that representing roles and fillers as vectors in this algebra yields a strict generalization of classical TPR\.

### C\.2Recursive Unbinding Theorem

Most proofs are Lean\-verified\.

###### Definition C\.1\(Vector Space Assumptions\)\.

LetVVbe a real inner product space with a symmetric, bilinear inner product denoted by\(⋅\)\(\\cdot\)\. We assume the standard left contraction \(interior product\) on the exterior algebra⋀V\\bigwedge V\.

###### Definition C\.2\(Contraction Convention\)\.

We adopt the standard convention that left contraction is left\-associative and evaluated sequentially from the inside out:

ar​⌟​…​⌟​a1​⌟​B≡ar​⌟​\(…​\(a1​⌟​B\)​…\)a\_\{r\}\\lrcorner\\dots\\lrcorner a\_\{1\}\\lrcorner B\\equiv a\_\{r\}\\lrcorner\(\\dots\(a\_\{1\}\\lrcorner B\)\\dots\)

###### Lemma C\.3\(Iterated Contraction Identity\)\.

LetU=\(u1,…,ur\)U=\(u\_\{1\},\\dots,u\_\{r\}\)be a sequence ofrrquery vectors andV=\(v1,…,vr\+1\)V=\(v\_\{1\},\\dots,v\_\{r\+1\}\)be a sequence ofr\+1r\+1target vectors forming a bladeB=v1∧⋯∧vr\+1B=v\_\{1\}\\wedge\\dots\\wedge v\_\{r\+1\}\. The iterated contraction ofUUontoBBis given by:

ur​⌟​\(…​\(u1​⌟​B\)​…\)=∑σ∈Sr\+1sgn​\(σ\)​\(∏ℓ=1ruℓ⋅vσ​\(ℓ\)\)​vσ​\(r\+1\)u\_\{r\}\\lrcorner\(\\dots\(u\_\{1\}\\lrcorner B\)\\dots\)=\\sum\_\{\\sigma\\in S\_\{r\+1\}\}\\mathrm\{sgn\}\(\\sigma\)\\left\(\\prod\_\{\\ell=1\}^\{r\}u\_\{\\ell\}\\cdot v\_\{\\sigma\(\\ell\)\}\\right\)v\_\{\\sigma\(r\+1\)\}
###### Proof\.

This is a standard result in exterior algebra relating iterated contractions to the generalized Laplace expansion of a determinant\(Doran and Lasenby,[2003](https://arxiv.org/html/2606.11391#bib.bib52)\)\.

While the contraction of a single vector introduces a position\-dependent sign\(−1\)m−1\(\-1\)^\{m\-1\}, the cumulative sign for a sequence ofrrcontractions corresponds exactly to the signature of the permutationσ\\sigma\. Specifically, there is a natural bijection between valid contraction paths \(sequences of distinct indices removed fromBB\) and the symmetric groupSr\+1S\_\{r\+1\}\. The permutationσ∈Sr\+1\\sigma\\in S\_\{r\+1\}is defined by mapping the contraction stepℓ\\ellto the target indexσ​\(ℓ\)\\sigma\(\\ell\), and mappingr\+1r\+1to the survivor indexσ​\(r\+1\)\\sigma\(r\+1\)\. The sign\(−1\)m−1\(\-1\)^\{m\-1\}for removing themm\-th vector accumulates multiplicatively overrrcontractions, resulting insgn​\(σ\)\\mathrm\{sgn\}\(\\sigma\)due to the parity of the permutation’s inversion count\. ∎

#### C\.2\.1Theorem Statement

###### Theorem C\.4\(Recursive Unbinding\)\.

Letr≥1r\\geq 1\. Consider a bundled objectAAconsisting ofppterms:

A=∑j=1paj1∧aj2∧⋯∧ajr∧fjA=\\sum\_\{j=1\}^\{p\}a\_\{j\_\{1\}\}\\wedge a\_\{j\_\{2\}\}\\wedge\\cdots\\wedge a\_\{j\_\{r\}\}\\wedge f\_\{j\}Notation:Letu=\(u1,…,ur\)=\(ai1,…,air\)u=\(u\_\{1\},\\dots,u\_\{r\}\)=\(a\_\{i\_\{1\}\},\\dots,a\_\{i\_\{r\}\}\)be the sequence of query vectors\. For each termjj, letv\(j\)=\(aj1,…,ajr,fj\)v^\{\(j\)\}=\(a\_\{j\_\{1\}\},\\ldots,a\_\{j\_\{r\}\},f\_\{j\}\)be the sequence of target vectors\.

The result of recursively unbindingAAusing the queryuuis:

air​⌟​\(⋯​\(ai1​⌟​A\)​⋯\)=∑j=1p\(dj​fj\+∑k=1rcjk​ajk\)a\_\{i\_\{r\}\}\\lrcorner\\left\(\\cdots\\left\(a\_\{i\_\{1\}\}\\lrcorner A\\right\)\\cdots\\right\)=\\sum\_\{j=1\}^\{p\}\\left\(d\_\{j\}f\_\{j\}\+\\sum\_\{k=1\}^\{r\}c\_\{j\_\{k\}\}a\_\{j\_\{k\}\}\\right\)where the coefficients are defined by sums over the symmetric groupSr\+1S\_\{r\+1\}:

dj\\displaystyle d\_\{j\}=∑σ∈Gr\+1\(sgn​σ\)​∏ℓ=1r\(uℓ⋅vσ​\(ℓ\)\(j\)\)\\displaystyle=\\sum\_\{\\sigma\\in G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(u\_\{\\ell\}\\cdot v^\{\(j\)\}\_\{\\sigma\(\\ell\)\}\)cjk\\displaystyle c\_\{j\_\{k\}\}=∑σ∈\(k,r\+1\)​Gr\+1\(sgn​σ\)​∏ℓ=1r\(uℓ⋅vσ​\(ℓ\)\(j\)\)\\displaystyle=\\sum\_\{\\sigma\\in\(k,r\+1\)G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(u\_\{\\ell\}\\cdot v^\{\(j\)\}\_\{\\sigma\(\\ell\)\}\)Here:

- •Gr\+1=Stab​\(r\+1\)≅SrG\_\{r\+1\}=\\mathrm\{Stab\}\(r\+1\)\\cong S\_\{r\}is the subgroup of permutations fixing indexr\+1r\+1\.
- •\(k,r\+1\)\(k,r\+1\)denotes the transposition swapping indiceskkandr\+1r\+1\.
- •\(k,r\+1\)​Gr\+1\(k,r\+1\)G\_\{r\+1\}is the left coset consisting of allσ\\sigmasuch thatσ​\(r\+1\)=k\\sigma\(r\+1\)=k\.

#### C\.2\.2Proof

###### Proof\.

By linearity of the contraction and the dot product, it suffices to prove the result for a single termB=v1∧⋯∧vr\+1B=v\_\{1\}\\wedge\\cdots\\wedge v\_\{r\+1\}\. For clarity, in the proof we drop the superscript\(j\)\(j\)when analyzing a single term, lettingv=\(aj1,…,ajr,fj\)v=\(a\_\{j\_\{1\}\},\\ldots,a\_\{j\_\{r\}\},f\_\{j\}\)\.

##### Step 1: Application of the Contraction Lemma\.

We apply Lemma[C\.3](https://arxiv.org/html/2606.11391#A3.Thmtheorem3)directly to the termBB\. The iterated contraction yields a summation over the symmetric groupSr\+1S\_\{r\+1\}:

Result=∑σ∈Sr\+1\(sgn​σ\)​\(∏ℓ=1ruℓ⋅vσ​\(ℓ\)\)​vσ​\(r\+1\)\\text\{Result\}=\\sum\_\{\\sigma\\in S\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\left\(\\prod\_\{\\ell=1\}^\{r\}u\_\{\\ell\}\\cdot v\_\{\\sigma\(\\ell\)\}\\right\)v\_\{\\sigma\(r\+1\)\}This step effectively expands the determinants of the interaction matrices, grouping terms by which vectorvσ​\(r\+1\)v\_\{\\sigma\(r\+1\)\}survives the contraction process\.

##### Step 2: Partitioning the Symmetric Group\.

We partition the sum based on the index of the surviving vector, determined byσ​\(r\+1\)\\sigma\(r\+1\)\. The set of indices is\{1,…,r\+1\}\\\{1,\\dots,r\+1\\\}\.

Case A: The filler survives \(σ​\(r\+1\)=r\+1\\sigma\(r\+1\)=r\+1\)\. The subset of permutations satisfying this condition is exactly the stabilizer subgroupGr\+1G\_\{r\+1\}\. For these terms, the surviving vector isvr\+1=fjv\_\{r\+1\}=f\_\{j\}\. The contribution to the sum is:

\(∑σ∈Gr\+1\(sgn​σ\)​∏ℓ=1r\(uℓ⋅vσ​\(ℓ\)\)\)​fj\\left\(\\sum\_\{\\sigma\\in G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(u\_\{\\ell\}\\cdot v\_\{\\sigma\(\\ell\)\}\)\\right\)f\_\{j\}This matches the definition ofdj​fjd\_\{j\}f\_\{j\}\.

Case B: A role survives \(σ​\(r\+1\)=k\\sigma\(r\+1\)=kfor somek∈\{1,…,r\}k\\in\\\{1,\\dots,r\\\}\)\. For a fixedkk, the set of permutations mappingr\+1r\+1tokkcorresponds to the left coset formed by composing the stabilizer with the transpositionτ=\(k,r\+1\)\\tau=\(k,r\+1\):

\{σ∈Sr\+1∣σ​\(r\+1\)=k\}=\(k,r\+1\)​Gr\+1\\\{\\sigma\\in S\_\{r\+1\}\\mid\\sigma\(r\+1\)=k\\\}=\(k,r\+1\)G\_\{r\+1\}This bijection ensures the coset\(k,r\+1\)​Gr\+1\(k,r\+1\)G\_\{r\+1\}exhaustively enumerates all permutations withσ​\(r\+1\)=k\\sigma\(r\+1\)=k, as left multiplication byτ\\tauis invertible\.

For these terms, the surviving vector isvk=ajkv\_\{k\}=a\_\{j\_\{k\}\}\. The contribution is:

∑k=1r\(∑σ∈\(k,r\+1\)​Gr\+1\(sgn​σ\)​∏ℓ=1r\(uℓ⋅vσ​\(ℓ\)\)\)​ajk\\sum\_\{k=1\}^\{r\}\\left\(\\sum\_\{\\sigma\\in\(k,r\+1\)G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(u\_\{\\ell\}\\cdot v\_\{\\sigma\(\\ell\)\}\)\\right\)a\_\{j\_\{k\}\}This matches the definition of∑cjk​ajk\\sum c\_\{j\_\{k\}\}a\_\{j\_\{k\}\}\.

##### Conclusion\.

Summing the results from Case A and Case B yields the expression in the theorem statement\. ∎

### C\.3Exact Recovery

###### Theorem C\.6\(Recursive Clifford Extension of TPR\)\.

Leta1,…,ara\_\{1\},\\ldots,a\_\{r\}be vectors inCl​\(n\+m\)\\mathrm\{Cl\}\(n\+m\)with support only on the firstnnbasis vectors, and letf1,…,fpf\_\{1\},\\ldots,f\_\{p\}be vectors with support only on the lastmmbasis vectors\. Assume the rolesaia\_\{i\}are orthonormal\. For a bundled objectA=∑j=1paj1∧aj2∧⋯∧ajr∧fjA=\\sum\_\{j=1\}^\{p\}a\_\{j\_\{1\}\}\\wedge a\_\{j\_\{2\}\}\\wedge\\cdots\\wedge a\_\{j\_\{r\}\}\\wedge f\_\{j\}, assume that the roles in each term are written in a canonical order \(e\.g\.,j1<j2<⋯<jrj\_\{1\}<j\_\{2\}<\\cdots<j\_\{r\}\)\. Then unbinding with the matching sequence of roles recovers the filler exactly:

air​⌟​\(air−1​⌟​\(⋯​\(ai1​⌟​A\)​⋯\)\)=fia\_\{i\_\{r\}\}\\lrcorner\\left\(a\_\{i\_\{r\-1\}\}\\lrcorner\\left\(\\cdots\\left\(a\_\{i\_\{1\}\}\\lrcorner A\\right\)\\cdots\\right\)\\right\)=f\_\{i\}whereiiis the unique index such that\(ai1,…,air\)=\(aj1,…,ajr\)\(a\_\{i\_\{1\}\},\\ldots,a\_\{i\_\{r\}\}\)=\(a\_\{j\_\{1\}\},\\ldots,a\_\{j\_\{r\}\}\), and the result is0if no suchjjexists\.

###### Proof\.

By the Recursive Unbinding theorem, with\(v1,…,vr\+1\)=\(aj1,…,ajr,fj\)\(v\_\{1\},\\ldots,v\_\{r\+1\}\)=\(a\_\{j\_\{1\}\},\\ldots,a\_\{j\_\{r\}\},f\_\{j\}\), we have

air​⌟​\(⋯​\(ai1​⌟​A\)​⋯\)=∑j=1p\(dj​fj\+∑k=1rcjk​ajk\)a\_\{i\_\{r\}\}\\lrcorner\\left\(\\cdots\\left\(a\_\{i\_\{1\}\}\\lrcorner A\\right\)\\cdots\\right\)=\\sum\_\{j=1\}^\{p\}\\left\(d\_\{j\}f\_\{j\}\+\\sum\_\{k=1\}^\{r\}c\_\{j\_\{k\}\}a\_\{j\_\{k\}\}\\right\)where

dj\\displaystyle d\_\{j\}=∑σ∈Gr\+1\(sgn​σ\)​∏ℓ=1r\(aiℓ⋅vσ​\(ℓ\)\)\\displaystyle=\\sum\_\{\\sigma\\in G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(a\_\{i\_\{\\ell\}\}\\cdot v\_\{\\sigma\(\\ell\)\}\)cjk\\displaystyle c\_\{j\_\{k\}\}=∑σ∈\(k,r\+1\)​Gr\+1\(sgn​σ\)​∏ℓ=1r\(aiℓ⋅vσ​\(ℓ\)\)\\displaystyle=\\sum\_\{\\sigma\\in\(k,r\+1\)G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(a\_\{i\_\{\\ell\}\}\\cdot v\_\{\\sigma\(\\ell\)\}\)
Fordjd\_\{j\}: sinceσ∈Gr\+1\\sigma\\in G\_\{r\+1\}fixesr\+1r\+1, we haveσ​\(ℓ\)∈\{1,…,r\}\\sigma\(\\ell\)\\in\\\{1,\\ldots,r\\\}for allℓ≤r\\ell\\leq r, so eachvσ​\(ℓ\)=ajσ​\(ℓ\)v\_\{\\sigma\(\\ell\)\}=a\_\{j\_\{\\sigma\(\\ell\)\}\}is a role\. Thus

dj=∑σ∈Gr\+1\(sgn​σ\)​∏ℓ=1r\(aiℓ⋅ajσ​\(ℓ\)\)d\_\{j\}=\\sum\_\{\\sigma\\in G\_\{r\+1\}\}\(\\mathrm\{sgn\}\\,\\sigma\)\\prod\_\{\\ell=1\}^\{r\}\(a\_\{i\_\{\\ell\}\}\\cdot a\_\{j\_\{\\sigma\(\\ell\)\}\}\)By orthonormality,aiℓ⋅ajσ​\(ℓ\)=δiℓ,jσ​\(ℓ\)a\_\{i\_\{\\ell\}\}\\cdot a\_\{j\_\{\\sigma\(\\ell\)\}\}=\\delta\_\{i\_\{\\ell\},j\_\{\\sigma\(\\ell\)\}\}\. The product is nonzero only ifiℓ=jσ​\(ℓ\)i\_\{\\ell\}=j\_\{\\sigma\(\\ell\)\}for allℓ\\ell\. Since both the query sequence\(i1,…,ir\)\(i\_\{1\},\\ldots,i\_\{r\}\)and the stored sequence\(j1,…,jr\)\(j\_\{1\},\\ldots,j\_\{r\}\)are in canonical order, the only permutation that can satisfy this is the identity\. Thusdj=1d\_\{j\}=1if\(i1,…,ir\)=\(j1,…,jr\)\(i\_\{1\},\\ldots,i\_\{r\}\)=\(j\_\{1\},\\ldots,j\_\{r\}\), anddj=0d\_\{j\}=0otherwise\.

Forcjkc\_\{j\_\{k\}\}: sinceσ∈\(k,r\+1\)​Gr\+1\\sigma\\in\(k,r\+1\)G\_\{r\+1\}sendsr\+1r\+1tokk, there exists someℓ\\ellwithσ​\(ℓ\)=r\+1\\sigma\(\\ell\)=r\+1, meaningvσ​\(ℓ\)=fjv\_\{\\sigma\(\\ell\)\}=f\_\{j\}\. Since roles and fillers have disjoint support,aiℓ⋅fj=0a\_\{i\_\{\\ell\}\}\\cdot f\_\{j\}=0, so the entire product vanishes\. Thuscjk=0c\_\{j\_\{k\}\}=0for allj,kj,k\.

Therefore

air​⌟​\(⋯​\(ai1​⌟​A\)​⋯\)=∑j=1pdj​fj=fia\_\{i\_\{r\}\}\\lrcorner\\left\(\\cdots\\left\(a\_\{i\_\{1\}\}\\lrcorner A\\right\)\\cdots\\right\)=\\sum\_\{j=1\}^\{p\}d\_\{j\}f\_\{j\}=f\_\{i\}whereiiis the index with\(ai1,…,air\)=\(aj1,…,ajr\)\(a\_\{i\_\{1\}\},\\ldots,a\_\{i\_\{r\}\}\)=\(a\_\{j\_\{1\}\},\\ldots,a\_\{j\_\{r\}\}\), and the result is0if no suchjjexists\. ∎

### C\.4Compressing TPR via Orthogonal Complements

The proofs above only use the disjoint support of roles and fillers to establish orthogonality\. This suggests a relaxation: if we instead require that fillers live in the orthogonal complement of the roles, exact recovery still holds, even when roles and fillers share the same basis vectors\.

This observation has implications for parameter efficiency\. Letting roles and fillers live on completely disjoint supports requiresn\|𝒞\|\+1n^\{\|\\mathcal\{C\}\|\+1\}parameters \(as it is TPR\)\. Letting them share all basis vectors while enforcing orthogonality reduces this to\(n\|𝒞\|\+1\)\\binom\{n\}\{\|\\mathcal\{C\}\|\+1\}parameters\. In other words, we can compress the TPR representation while keeping it lossless by constraining fillers to the orthogonal complement of the role subspace—precisely the geometric relationship that OSC enforces through subspace carving\.

## Appendix DExperiments

### D\.1Extended Experimental Results

Prior to presenting more experimental results, we outline key implementation details\. Context generation requires QR decomposition of ad×kd\\times kmatrix costing𝒪​\(d​k2\)\\mathcal\{O\}\(dk^\{2\}\)operations, independent of context size\. Simple testing shows thatkkclose tod−dd\-\\sqrt\{d\}is optimal\. Therefore, instead of computing the carving operator through the projection matrix, we compute the operator directly, generating onlyk1=d−kk\_\{1\}=d\-korthogonal vectors\. This is negligible fork1≪dk\_\{1\}\\ll dand extremely parallelizable across contexts \(simply outer products and additions\)\. Frequently accessed contexts can be cached\.

##### Other baseline VSA experiments\.

We give more experiments\. Figure[6](https://arxiv.org/html/2606.11391#A4.F6)shows OSC’s performance against VSAs of dimensiond=8100d=8100\. To showcase the fact that OSC is invariant to context size, we also perform the same experiment for role depth of4848and compare against VSAs with dimensiond=8100d=8100in Figure[7](https://arxiv.org/html/2606.11391#A4.F7)\. This dimension was chosen as it is a perfect square, which required by VTB, that is close to81928192\. Figure[8](https://arxiv.org/html/2606.11391#A4.F8)shows OSC’s performance withp=2p=2against VSA when superposition memory is held constant\. As with Figure[5](https://arxiv.org/html/2606.11391#S6.F5), OSC beats the VSA with the highest dimension with fewer params that the VSA with the lowest dimension\. We presented retrieval accuracy for OSC withp=3p=3in Figure[5](https://arxiv.org/html/2606.11391#S6.F5)in §[6](https://arxiv.org/html/2606.11391#S6); we present both the retrieval and recognition accuracy in Figure[9](https://arxiv.org/html/2606.11391#A4.F9)\.

![Refer to caption](https://arxiv.org/html/2606.11391v1/x5.png)
![Refer to caption](https://arxiv.org/html/2606.11391v1/x6.png)

Figure 6:Performance comparison of1414VSAs atd=8100d=8100and OSC acrossd=25,50,75,100,125,150,175,d=25,50,75,100,125,150,175,and200200forp=2p=2at increasing bundle counts with unique fillers and role depth11\. OSC outperforms all VSAs on both retrieval and recognition tasks\. Note the difference inYYaxes between graphs\. Standard deviation is shown as the shaded regions\.![Refer to caption](https://arxiv.org/html/2606.11391v1/x7.png)
![Refer to caption](https://arxiv.org/html/2606.11391v1/x8.png)

Figure 7:Performance comparison of1414VSAs atd=8100d=8100and OSC acrossd=25,50,75,100,125,150,175,d=25,50,75,100,125,150,175,and200200forp=2p=2at increasing bundle counts with unique fillers and role depth4848\. OSC outperforms all VSAs on both retrieval and recognition tasks\. Note the difference inYYaxes between graphs\. Standard deviation is shown as the shaded regions\. OSC’s performance for a role depth of4848is the exact same as for a depth of11\(Figure[6](https://arxiv.org/html/2606.11391#A4.F6)\)\.![Refer to caption](https://arxiv.org/html/2606.11391v1/x9.png)
![Refer to caption](https://arxiv.org/html/2606.11391v1/x10.png)

Figure 8:Performance of HLB at varying dimensions against OSC forp=2p=2\. Identically colored runs share the same superposition memory size \(not total memory\)\. VSAs are slightly superior than OSC in memory capacity, but OSC’s sub\-linear scaling requires the VSA to have2002∗1001/\(2002\+2∗200∗1000\)=91×200^\{2\}\*1001/\\left\(200^\{2\}\+2\*200\*1000\\right\)=91\\timesOSC’s total memory to get close in retrieval accuracy at10001000bundles\. Standard deviations are denoted by the shaded regions\.![Refer to caption](https://arxiv.org/html/2606.11391v1/x11.png)
![Refer to caption](https://arxiv.org/html/2606.11391v1/x12.png)

Figure 9:Performance of HLB at varying dimensions against OSC forp=3p=3\. Identically colored runs share the same superposition memory size \(not total memory\)\. VSAs are slightly superior than OSC in memory capacity, but OSC’s sub\-linear scaling requires the VSA to have703∗10001/\(803\+3∗80∗10000\)≈1178×70^\{3\}\*10001/\\left\(80^\{3\}\+3\*80\*10000\\right\)\\approx 1178\\timesOSC’s total memory to achieve comparable retrieval accuracy\. Standard deviations \(shaded regions\) are present but negligible\.![Refer to caption](https://arxiv.org/html/2606.11391v1/x13.png)Figure 10:Adapted from Figure[4](https://arxiv.org/html/2606.11391#S6.F4), this plot excludes VSAs to highlight the logarithmic relationship between the filler dimension and codebook size for OSC\. Minimum filler dimension required to achieve99%99\\%accuracy\. A dimension was selected only if mean accuracy across 10 trials exceeded99%99\\%at that dimension but fell below99%99\\%at the next lowest dimension\. OSC exhibits the same favorable logarithmic scaling with codebook size as VSAs\. The dotted line tracks the actual dimension points, while the solid line represents the best fit\.

### D\.2Discriminative Recognition vs Generative Retrieval

We evaluated OSC retrieval with recognition\. To determine if this methodology impacts VSA performance, we applied the same recognition\-based retrieval to FHRR, HLB, MAP\_I, and VTB\. For FHRR, HLB, and MAP\_I, binding and unbinding are adjoint operations, so⟨Unbind​\(M,r\),g⟩=⟨M,bind​\(r,g\)⟩\\langle\\text\{Unbind\}\(M,r\),g\\rangle=\\langle M,\\text\{bind\}\(r,g\)\\ranglefor every candidategg, and the two methods return the samearg⁡max\\arg\\max\. VTB does not satisfy this adjoint relation, but we observed no significant difference between standard retrieval and recognition\-based retrieval at dimensionsd=4096d=4096andd=8100d=8100\. As shown in Figure[11](https://arxiv.org/html/2606.11391#A4.F11), the accuracy closely mirrors the results in Figures[3](https://arxiv.org/html/2606.11391#S5.F3)and[6](https://arxiv.org/html/2606.11391#A4.F6)\.

![Refer to caption](https://arxiv.org/html/2606.11391v1/x14.png)
![Refer to caption](https://arxiv.org/html/2606.11391v1/x15.png)

Figure 11:Retrieval accuracy when using the recognition\-based retrieval approach for FHRR, HLB, MAP\_I, and VTB\. The curves ford=4096d=4096andd=8100d=8100match those of Figure[3](https://arxiv.org/html/2606.11391#S5.F3)and Figure[6](https://arxiv.org/html/2606.11391#A4.F6)\. Shaded regions denote standard deviations\.
### D\.3Carving dimension\.

We investigate the optimal carving dimensionk1k\_\{1\}into which filler components are projected\. The trade\-off involves two competing terms, cross\-context interference \(favoring largek1k\_\{1\}\) and within\-context filler discrimination \(favoring smallk1k\_\{1\}\)\. Empirically, we find that scaling the carving dimension with𝒪​\(d\)\\mathcal\{O\}\(\\sqrt\{d\}\)to be optimal\. While further analysis is needed to determine if the true optimum scales as𝒪​\(d1p\)\\mathcal\{O\}\\left\(d^\{\\frac\{1\}\{p\}\}\\right\), the practical distinction is negligible in our regime\. The flexibility of the tensor orderppallows us to operate with relatively small filler dimensions, where a broad range of subspace sizes yield stable results\.

![Refer to caption](https://arxiv.org/html/2606.11391v1/x16.png)Figure 12:Heatmap of retrieval accuracy for filler dimensions ranging from5050to200200in increments of1010\. Accuracy is normalized per dimension, with the maximum value scaled to11\.
### D\.4FLOP Analysis

We complement the wall\-clock measurements in Table[3](https://arxiv.org/html/2606.11391#S6.T3)with a FLOP analysis of the retrieval operation\. Counts are derived from the matrix dimensions in the timed code\.

##### HLB\.

Retrieval consists of an elementwise division to unbind the role and a single matrix\-vector product against the candidate codebook\. The dominant cost is

FLOPsHLB≈2​L​d,\\mathrm\{FLOPs\}\_\{\\mathrm\{HLB\}\}\\approx 2Ld,\(34\)whereLLis the vocabulary size andddis the dimension\.

##### OSC\.

Retrieval consists of three steps\. First, the complement role\-subspace projectorP=r⊤​rP=r^\{\\top\}ris formed, costing2​k1​d22k\_\{1\}d^\{2\}wherek1k\_\{1\}is the carving dimension\. Second, the candidates are projected throughPP, costing2​L​p​d22Lpd^\{2\}\. Note that rejecting onto the orthogonal complement is the same as projecting onto the subspace\. Third, the projected candidates are contracted against the order\-ppmemory tensor, costing2​L​dp2Ld^\{p\}\. The dominant cost is

FLOPsO​S​C≈2​k1​d2\+2​L​p​d2\+2​L​dp\.\\mathrm\{FLOPs\}\_\{OSC\}\\approx 2k\_\{1\}d^\{2\}\+2Lpd^\{2\}\+2Ld^\{p\}\.\(35\)

##### Benchmark FLOP counts\.

Table[5](https://arxiv.org/html/2606.11391#A4.T5)reports total FLOPs for the three configurations of Table[3](https://arxiv.org/html/2606.11391#S6.T3), usingk1=⌊d⌋k\_\{1\}=\\lfloor\\sqrt\{d\}\\rflooras in the timing code\.B1: OSC \(d=64,p=2d=64,p=2\) vs HLB \(d=4,096d=4\{,\}096\)\.B2: OSC \(d=200,p=2d=200,p=2\) vs HLB \(d=40,000d=40\{,\}000\)\.B3: OSC \(d=75,p=3d=75,p=3\) vs HLB \(d=421,875d=421\{,\}875\)\.

Table 5:Total FLOPs for HLB vs\. OSC retrieval at the configurations of Table[3](https://arxiv.org/html/2606.11391#S6.T3)\. OSC’s FLOP count exceeds HLB’s at every configuration, yet wall\-clock timing favors OSC at scale because the bottleneck is memory bandwidth rather than arithmetic throughput\. When OSC hasp=2p=2, the FLOPs count is≈3×\\approx 3\\timesHLB, while when OSC hasp=3p=3the FLOPs count is approximately equal due to the very large filler dimension\.

## Appendix EHyperparameters

Table[6](https://arxiv.org/html/2606.11391#A5.T6)contains the hyperparameters for the XML experiments\. Figure[12](https://arxiv.org/html/2606.11391#A4.F12)discusses the optimal carving dimension based on the filler dimension of OSC\.

Table 6:Complete hyperparameter configuration\. Expansion refers to the width multiplier for the second hidden layer \(e\.g\.,512×2=1024512\\times 2=1024for Eurlex\)\. Abbreviations: BS \(Batch Size\), Drop \(Dropout Rate\), Output Dim \(Output dimension for HLB/VTB/MAP\), OSC d \(Filler dimension for OSC\)\. Our implementation builds upon the codebase ofAlamet al\.\([2024](https://arxiv.org/html/2606.11391#bib.bib20)\)\. We utilize the optimal hyperparameters reported in their work\. When running their code, we could not recreate HLB’s dominance\. We contacted the authors, but could not resolve the issue\.