Vector Symbolic Policy Gradient

# Vector Symbolic Policy Gradient
Source: [https://arxiv.org/html/2608.18404](https://arxiv.org/html/2608.18404)
Sanggeon YunAffiliation:University of California, IrvineSungHeon JeongAffiliation:University of California, IrvineHyunwoo OhAffiliation:University of California, IrvineRaheeb HassanAffiliation:University of California, IrvinePietro MercatiAffiliation:Intel CorporationNathaniel D\. BastianAffiliation:Johns Hopkins UniversityMahdi ImaniAffiliation:Northeastern UniversityMohsen ImaniAffiliation:University of California, Irvine

###### Abstract

Vector Symbolic Architecture \(VSA\) is built around a simple idea: distributed memories can be learned through lightweight algebra and remain useful even when their bits are unreliable\. Yet this perspective has rarely been connected directly to discrete\-action policy gradients\. We introduce Vector\-Symbolic Policy Gradient \(VSPG\), a categorical actor that represents each action by a unit\-norm hypervector and chooses actions by similarity to an encoded state\. We show that the standard softmax policy\-gradient step has an exact vector\-symbolic interpretation: advantage\-weighted state hypervectors are bundled into the selected action memory and suppressed in competing memories, followed by row\-wise normalization, so the actor trains in closed form with no optimizer state and logits bounded by construction\. Over training, these memories become fixed\-size compressed kernel expansions that transfer advantage evidence across similar states without retaining past samples at inference\. On classic control, MiniGrid, and multi\-agent SustainGym, VSPG achieves competitive returns with favorable learning speed\. Its distributed action memories also degrade substantially more gracefully than neural and linear actors under post\-training quantization and random bit flips, making VSPG a promising actor for unreliable edge systems\. An anonymized code is available[here](https://github.com/BiasLabProjects/VSPG.git)\.

## 1Introduction

Vector Symbolic Architecture \(VSA\), also known as Hyperdimensional computing \(HDC\), is a brain\-inspired computing paradigm rooted in theories of distributed representation from cognitive science\[[12](https://arxiv.org/html/2608.18404#bib.bib43),[26](https://arxiv.org/html/2608.18404#bib.bib7)\]\. VSA represents data using high\-dimensional distributed vectors, or hypervectors, and performs computation through simple operations such as similarity search, bundling, binding, and normalization\.

Two properties are crucial for using VSA as a learning representation\. First, independently generated hypervectors are nearly orthogonal in high dimension, allowing many pieces of information to be superposed in a single memory with limited interference and supporting graceful degradation under noise\[[13](https://arxiv.org/html/2608.18404#bib.bib22),[8](https://arxiv.org/html/2608.18404#bib.bib23)\]\. Second, an VSA encoderϕ\\phimaps inputs into an explicit high\-dimensional inner\-product space, whereϕ​\(x\)⊤​ϕ​\(x′\)\\phi\(x\)^\{\\top\}\\phi\(x^\{\\prime\}\)can be designed to preserve a meaningful similarity between inputs\. This gives VSA encoders a theoretically grounded connection to kernel\-approximation methods\[[33](https://arxiv.org/html/2608.18404#bib.bib47),[46](https://arxiv.org/html/2608.18404#bib.bib6)\]\. Together, these properties have motivated the use of VSA in lightweight and resource\-constrained learning systems, with recent work spanning intelligent sensing, hardware acceleration of IoT devices\[[42](https://arxiv.org/html/2608.18404#bib.bib16),[20](https://arxiv.org/html/2608.18404#bib.bib17),[3](https://arxiv.org/html/2608.18404#bib.bib18),[5](https://arxiv.org/html/2608.18404#bib.bib19),[2](https://arxiv.org/html/2608.18404#bib.bib20)\]\.

![Refer to caption](https://arxiv.org/html/2608.18404v1/teaser.png)Figure 1:Overview of VSPG\. A fixed encoder maps the observation to a hypervector, action hypervectors score it by inner products, and the exact policy\-gradient step bundles the encoded state into the kernel action memories\.These properties are particularly relevant to deployment robustness in reinforcement learning \(RL\), where practical agents may run under low\-precision execution or unreliable memory, and bit\-level corruption of stored policy parameters can degrade autonomous decision making\[[7](https://arxiv.org/html/2608.18404#bib.bib29),[36](https://arxiv.org/html/2608.18404#bib.bib26),[37](https://arxiv.org/html/2608.18404#bib.bib28)\]\. VSA provides a natural representation\-level substrate for lightweight RL policies exposed to quantization and model\-state faults\. Existing VSA\-based RL methods have mostly used hypervectors at the algorithmic and application level, either as Q\-value approximators trained by Bellman regression\[[23](https://arxiv.org/html/2608.18404#bib.bib2),[24](https://arxiv.org/html/2608.18404#bib.bib3)\]or as components of continuous actor–critic control\[[25](https://arxiv.org/html/2608.18404#bib.bib1),[14](https://arxiv.org/html/2608.18404#bib.bib46)\], but leave open a basic question: can action hypervectors directly parameterize a categorical actor, and can its policy\-gradient update be written as a vector\-symbolic memory operation?

Table 1:Positioning of VSPG among the method families it builds on\.We answer this question with*Vector\-Symbolic Policy Gradient*\(VSPG\), a discrete\-action actor that represents each action by a unit\-norm hypervector and scores it by similarity to the encoded state\. Under the standard softmax policy\-gradient surrogate, we prove that its update is exactly advantage\-weighted hypervector bundling followed by normalization, and therefore supports standard advantage estimators\[[29](https://arxiv.org/html/2608.18404#bib.bib5),[40](https://arxiv.org/html/2608.18404#bib.bib25)\]\. We further show that each trained action hypervector is a fixed\-size compressed kernel memory, storing an advantage\-weighted kernel expansion over visited states and transferring evidence according to the encoder\-induced similarity\. This provides a concrete mechanism that can support sample\-efficient learning without increasing inference\-time memory\. Finally, for bipolar action memories, we prove that greedy action selection is stable under random bit flips, with failure probability decaying exponentially in the hypervector dimension\. VSPG thus connects VSA action memories, log\-linear policy gradients, and kernel policy search while providing a quantitative robustness guarantee\.

We evaluate VSPG on classic control, MiniGrid\[[34](https://arxiv.org/html/2608.18404#bib.bib37),[6](https://arxiv.org/html/2608.18404#bib.bib38)\], and multi\-agent SustainGym building control\[[38](https://arxiv.org/html/2608.18404#bib.bib40)\], against DNN and linear actor baselines, as well as QHD\[[23](https://arxiv.org/html/2608.18404#bib.bib2)\], a strong value\-based VSA method for discrete\-action RL\. Across these benchmarks, VSPG achieves competitive final performance and stronger sample efficiency than matched actor baselines\. Under post\-training bit\-flip corruption, its distributed action memories also retain performance substantially better than both DNN and raw linear actors, demonstrating graceful degradation under model\-state faults\.

Our contributions are as follows\.

- •We introduce VSPG, a discrete\-action policy\-gradient actor that represents each action with a unit\-norm hypervector, and show that its update admits an exact vector\-symbolic interpretation as advantage\-weighted bundling followed by row normalization\.
- •We show that trained action hypervectors form fixed\-size compressed kernel memories and prove robustness of bipolar action memories to random bit flips\.
- •We evaluate VSPG on classic control, MiniGrid, and multi\-agent building control against neural, linear and value\-based VSA, demonstrating competitive performance, favorable sample efficiency, and robustness to quantization and bit\-level faults\.

## 2Related Work

### 2\.1VSA and Kernel Policies for RL

Prior VSA\-RL methods mainly use hypervectors as lightweight function approximators within existing RL formulations\. HDPG\[[25](https://arxiv.org/html/2608.18404#bib.bib1)\]addresses continuous control with VSA\-based Gaussian actors and critics, whereas discrete\-action methods largely follow the value\-based QHD framework\[[23](https://arxiv.org/html/2608.18404#bib.bib2),[24](https://arxiv.org/html/2608.18404#bib.bib3)\], learning action\-specificQQ\-value hypervectors through Bellman\-error\-weighted bundling\. This value\-based line has also been applied to cybersecurity, robotics, navigation, and sensing applications\[[10](https://arxiv.org/html/2608.18404#bib.bib8),[16](https://arxiv.org/html/2608.18404#bib.bib9),[17](https://arxiv.org/html/2608.18404#bib.bib4),[18](https://arxiv.org/html/2608.18404#bib.bib39)\]\. VSPG instead directly parameterizes a categorical softmax policy with action hypervectors, extending vector\-symbolic learning from value approximation to discrete\-action policy gradients\. VSA methods have also demonstrated favorable data efficiency in single\-pass and online learning\[[9](https://arxiv.org/html/2608.18404#bib.bib42),[41](https://arxiv.org/html/2608.18404#bib.bib15)\], and prior VSA\-RL studies report promising learning efficiency, with NavHD\[[17](https://arxiv.org/html/2608.18404#bib.bib4)\]providing systematic multi\-seed evidence in robotic navigation; VSPG examines whether this behavior extends to discrete\-action policy gradients\. VSPG further connects to kernelized and log\-linear policy search: kernel policies represent action scores as expansions over experience\[[11](https://arxiv.org/html/2608.18404#bib.bib14),[19](https://arxiv.org/html/2608.18404#bib.bib12),[45](https://arxiv.org/html/2608.18404#bib.bib13)\], whereas VSPG superposes this expansion into one fixed\-size hypervector per action, requiring only one inner product per action at inference\. With a fixed encoder, VSPG is a log\-linear softmax policy over random features\[[28](https://arxiv.org/html/2608.18404#bib.bib33),[32](https://arxiv.org/html/2608.18404#bib.bib10)\], with optimization covered by established policy\-gradient theory\[[21](https://arxiv.org/html/2608.18404#bib.bib31),[1](https://arxiv.org/html/2608.18404#bib.bib32)\]\. As summarized in[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1), VSPG connects VSA\-based RL, log\-linear policy gradients, and kernel policy search through an exact VSPG update, a fixed\-size kernel\-memory expansion, and a stability guarantee under bit\-level corruption\.

### 2\.2Robustness to Model\-State Corruption in RL

RL policies deployed on resource\-constrained systems may be affected by corruption of the stored policy itself\[[39](https://arxiv.org/html/2608.18404#bib.bib41),[44](https://arxiv.org/html/2608.18404#bib.bib36),[7](https://arxiv.org/html/2608.18404#bib.bib29)\]\. Low\-voltage operation and approximate memory can introduce bit\-level errors in model parameters, reducing the reliability of autonomous decisions\. Prior work mainly addresses this problem through fault\-aware training, bit\-error injection, or hardware\-level adaptation for neural policies\[[15](https://arxiv.org/html/2608.18404#bib.bib27),[37](https://arxiv.org/html/2608.18404#bib.bib28),[36](https://arxiv.org/html/2608.18404#bib.bib26)\]\. VSPG instead takes a complementary representation\-level approach, storing a discrete\-action policy as distributed vector\-symbolic action memories\. We evaluate degradation under quantization and model\-state bit corruption, while our theoretical analysis provides an exponential failure bound in the hypervector dimension for bipolar memories at a fixed similarity margin \([Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1)\)\.

## 3Preliminaries

### 3\.1Reinforcement Learning and Policy Gradients

We formulate the problem as a decentralized partially observable Markov decision process \(Dec\-POMDP\), which includes fully observed single\-agent MDPs as a special case\. A Dec\-POMDP is defined asℳ=\(ℐ,𝒮,\{𝒜i\}i∈ℐ,P,r,ρ0,\{𝒳i\}i∈ℐ,O,γ\),\\mathcal\{M\}=\\bigl\(\\mathcal\{I\},\\mathcal\{S\},\\\{\\mathcal\{A\}\_\{i\}\\\}\_\{i\\in\\mathcal\{I\}\},P,r,\\rho\_\{0\},\\\{\\mathcal\{X\}\_\{i\}\\\}\_\{i\\in\\mathcal\{I\}\},O,\\gamma\\bigr\),whereℐ\\mathcal\{I\}is the set of agents,𝒮\\mathcal\{S\}is the state space,𝒜i\\mathcal\{A\}\_\{i\}and𝒳i\\mathcal\{X\}\_\{i\}are the action and observation spaces of agentii,PPis the state transition probability,rris the shared reward function,ρ0\\rho\_\{0\}is the initial\-state distribution,OOare the observations, andγ∈\[0,1\]\\gamma\\in\[0,1\]is the discount factor\. At timett, the environment is in statests\_\{t\}, each agent receives observationxtix\_\{t\}^\{i\}, and selectsati∼πθi\(⋅∣xti\)\.a\_\{t\}^\{i\}\\sim\\pi\_\{\\theta\_\{i\}\}\(\\cdot\\mid x\_\{t\}^\{i\}\)\.The joint action𝐚t=\(ati\)i∈ℐ\\mathbf\{a\}\_\{t\}=\(a\_\{t\}^\{i\}\)\_\{i\\in\\mathcal\{I\}\}determines the subsequent transition and reward\. Let𝜽=\{θi\}i∈ℐ\\boldsymbol\{\\theta\}=\\\{\\theta\_\{i\}\\\}\_\{i\\in\\mathcal\{I\}\}denote the joint policy parameters\. The objective is to maximize the expected discounted return

J⁡\(𝜽\)=𝔼τ∼𝝅𝜽​\[∑t=0T−1γt​rt\],J\(\\boldsymbol\{\\theta\}\)=\\mathbb\{E\}\_\{\\tau\\sim\\boldsymbol\{\\pi\}\_\{\\boldsymbol\{\\theta\}\}\}\\left\[\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{t\}r\_\{t\}\\right\],\(1\)whereτ\\tauis a trajectory induced by the joint policy and environment\. For a factorized decentralized policy, the policy gradient for agentiiis

∇θiJ​\(𝜽\)=𝔼𝝅𝜽​\[∑t=0T−1A^ti​∇θi​log⁡πθi​\(ati∣xti\)\],\\nabla\_\{\\theta\_\{i\}\}J\(\\boldsymbol\{\\theta\}\)=\\mathbb\{E\}\_\{\\boldsymbol\{\\pi\}\_\{\\boldsymbol\{\\theta\}\}\}\\left\[\\sum\_\{t=0\}^\{T\-1\}\\hat\{A\}\_\{t\}^\{i\}\\nabla\_\{\\theta\_\{i\}\}\\log\\pi\_\{\\theta\_\{i\}\}\(a\_\{t\}^\{i\}\\mid x\_\{t\}^\{i\}\)\\right\],\(2\)whereA^ti\\hat\{A\}\_\{t\}^\{i\}is an advantage estimate\[[31](https://arxiv.org/html/2608.18404#bib.bib44),[32](https://arxiv.org/html/2608.18404#bib.bib10)\], obtained from returns or an actor–critic estimator such as Generalized Advantage Estimation \(GAE\)\[[14](https://arxiv.org/html/2608.18404#bib.bib46),[29](https://arxiv.org/html/2608.18404#bib.bib5),[30](https://arxiv.org/html/2608.18404#bib.bib45),[40](https://arxiv.org/html/2608.18404#bib.bib25)\]\. VSPG uses the same policy\-gradient objective but represents each categorical actor with action hypervectors, yielding an advantage\-weighted bundling update that applies to both single\- and multi\-agent settings\.

### 3\.2VSA Basics

VSA encodes inputs as high\-dimensional hypervectors and computes through bundling \(superposition\), binding, and similarity search\. Although bundling and binding can be implemented in different ways, in this paper they refer to element\-wise addition and element\-wise multiplication, respectively\. Let𝐡=φ⁡\(x\)∈ℝD\\mathbf\{h\}=\\varphi\(x\)\\in\\mathbb\{R\}^\{D\}denote the hypervector produced by a fixed encoderφ\\varphi, normalized when used for similarity search\. Random hypervectors are quasi\-orthogonal, allowing many items to be bundled with limited interference\.

###### Lemma 1\(Quasi\-orthogonality\)\.

Let𝐮,𝐯∈ℝD\\mathbf\{u\},\\mathbf\{v\}\\in\\mathbb\{R\}^\{D\}be independent random unit vectors, at least one of which is uniformly distributed on the sphere\. Then𝔼⁡\[𝐮⊤​𝐯\]=0\\mathbb\{E\}\[\\mathbf\{u\}^\{\\top\}\\mathbf\{v\}\]=0and, for everyδ∈\(0,1\)\\delta\\in\(0,1\),

Pr\[\|𝐮⊤𝐯\|≥2​ln⁡\(2/δ\)D\]≤δ\.\\Pr\\Bigl\[\\,\|\\mathbf\{u\}^\{\\top\}\\mathbf\{v\}\|\\geq\\sqrt\{\\tfrac\{2\\ln\(2/\\delta\)\}\{D\}\}\\,\\Bigr\]\\leq\\delta\.

###### Proof\.

Condition on𝐯\\mathbf\{v\}\. By rotational invariance,𝐮⊤​𝐯\\mathbf\{u\}^\{\\top\}\\mathbf\{v\}is distributed as one coordinate of a unit vector, whose spherical\-cap measure satisfiesPr\[\|u1\|≥t\]≤2e−Dt2/2\\Pr\[\|u\_\{1\}\|\\geq t\]\\leq 2e^\{\-Dt^\{2\}/2\}\[[35](https://arxiv.org/html/2608.18404#bib.bib30)\]; the tail bound and zero mean follow\. ∎

This property enables associative memory by bundling many examples into a single prototype\. A standard VSA classifier stores one prototype per class,

𝐂ℓ=∑i:yi=ℓwiϕ\(xi\)\.\\mathbf\{C\}\_\{\\ell\}=\\sum\_\{i:y\_\{i\}=\\ell\}w\_\{i\}\\phi\(x\_\{i\}\)\.\(3\)Prototypes superpose class examples, optionally weighted\[[9](https://arxiv.org/html/2608.18404#bib.bib42)\], and predict by similarity search\.y^=arg⁡maxℓ⁡δ⁡\(ϕ⁡\(xq\),𝐂ℓ\)\\hat\{y\}=\\arg\\max\_\{\\ell\}\\delta\\\!\\left\(\\phi\(x\_\{q\}\),\\mathbf\{C\}\_\{\\ell\}\\right\), whereδ\\deltais an inner\-product\-based similarity such as cosine similarity\. Quasi\-orthogonality limits interference, while repeated or similar components reinforce\. Under inner\-product scoring, prototypes are linear weights over fixed hypervector features and may be formed by bundling or gradient\-based optimization\[[41](https://arxiv.org/html/2608.18404#bib.bib15)\]\. VSPG applies this view to discrete\-action policies: action hypervectors serve as policy weights, encoded states as fixed features, and policy\-gradient learning becomes advantage\-weighted bundling, yielding compressed kernel memories over visited states\.

## 4Vector\-Symbolic Policy Gradient

Algorithm 1Vector\-Symbolic Policy Gradient \(VSPG\)0:Fixed normalized encoder

ϕ\\boldsymbol\{\\phi\}, action HVs

𝐂=\{𝐜a\}a∈𝒜\\mathbf\{C\}=\\\{\\mathbf\{c\}\_\{a\}\\\}\_\{a\\in\\mathcal\{A\}\}, temperature

τ\\tau, learning rate

η\\eta
1:Initialize each

𝐜a\\mathbf\{c\}\_\{a\}and rescale to

‖𝐜a‖2=1\\\|\\mathbf\{c\}\_\{a\}\\\|\_\{2\}=1
2:foreach batch of episodesdo

3:Collect transitions using

𝐬t=ϕ⁡\(xt\)\\mathbf\{s\}\_\{t\}=\\boldsymbol\{\\phi\}\(x\_\{t\}\)and

π⁡\(a∣xt\)=softmaxa⁡\(τ​𝐜a⊤​𝐬t\)\\pi\(a\\mid x\_\{t\}\)=\\operatorname\{softmax\}\_\{a\}\\\!\\bigl\(\\tau\\,\\mathbf\{c\}\_\{a\}^\{\\top\}\\mathbf\{s\}\_\{t\}\\bigr\)
4:Estimate advantages

\{At\}\\\{A\_\{t\}\\\}
5:

Λt,a←Atτ\(𝟏\[a=at\]−π\(a∣xt\)\)\\Lambda\_\{t,a\}\\leftarrow A\_\{t\}\\,\\tau\\bigl\(\\mathbf\{1\}\[a=a\_\{t\}\]\-\\pi\(a\\mid x\_\{t\}\)\\bigr\)
6:

𝐂←𝐂\+η​𝚲⊤​𝐒\\mathbf\{C\}\\leftarrow\\mathbf\{C\}\+\\eta\\,\\boldsymbol\{\\Lambda\}^\{\\top\}\\mathbf\{S\}
7:

𝐜a←𝐜a/‖𝐜a‖2\\mathbf\{c\}\_\{a\}\\leftarrow\\mathbf\{c\}\_\{a\}/\\\|\\mathbf\{c\}\_\{a\}\\\|\_\{2\}for each

a∈𝒜a\\in\\mathcal\{A\}
8:endfor

### 4\.1Policy Representation and Update

VSPG turns the VSA prototype classifier of the preliminaries into a stochastic policy: class prototypes become action hypervectors, and similarity scores become policy logits\. Throughout,xtx\_\{t\}denotes the policy input at timett— the state in fully observed tasks and the observation otherwise\. Every encoder normalizes its raw outputϕ~​\(x\)∈ℝD\\tilde\{\\boldsymbol\{\\phi\}\}\(x\)\\in\\mathbb\{R\}^\{D\}before it reaches the actor,

ϕ⁡\(x\)=ϕ~​\(x\)‖ϕ~​\(x\)‖2,‖ϕ⁡\(x\)‖2=1\.\\boldsymbol\{\\phi\}\(x\)=\\frac\{\\tilde\{\\boldsymbol\{\\phi\}\}\(x\)\}\{\\\|\\tilde\{\\boldsymbol\{\\phi\}\}\(x\)\\\|\_\{2\}\},\\qquad\\\|\\boldsymbol\{\\phi\}\(x\)\\\|\_\{2\}=1\.\(4\)We write𝐬t=ϕ⁡\(xt\)\\mathbf\{s\}\_\{t\}=\\boldsymbol\{\\phi\}\(x\_\{t\}\)and stack a batch ofTTencoded inputs as𝐒∈ℝT×D\\mathbf\{S\}\\in\\mathbb\{R\}^\{T\\times D\}\. The actor parameters are the action hypervectors𝐂∈ℝ\|𝒜\|×D\\mathbf\{C\}\\in\\mathbb\{R\}^\{\|\\mathcal\{A\}\|\\times D\}, with one unit\-norm row𝐜a\\mathbf\{c\}\_\{a\}per action\. Unless otherwise stated, each row is initialized independently from an isotropic Gaussian distribution and normalized to unit norm, independently of the fixed encoder\. The policy scores each action by an inner product and takes a softmax,

π⁡\(a∣x\)=softmaxa⁡\(τ​𝐂​ϕ​\(x\)\),\\pi\(a\\mid x\)=\\operatorname\{softmax\}\_\{a\}\\\!\\bigl\(\\tau\\,\\mathbf\{C\}\\,\\boldsymbol\{\\phi\}\(x\)\\bigr\),\(5\)so, both factors being unit\-norm, each logit is a bounded scaled cosine,

ℓa​\(x\)=τ​𝐜a⊤​ϕ​\(x\)∈\[−τ,τ\],\\ell\_\{a\}\(x\)=\\tau\\,\\mathbf\{c\}\_\{a\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)\\in\[\-\\tau,\\tau\],\(6\)andτ\>0\\tau\>0sets how sharp the policy can be; a batch is scored in the single matrix productτ​𝐒𝐂⊤\\tau\\,\\mathbf\{S\}\\mathbf\{C\}^\{\\top\}\. Because both factors are unit norm, each logit lies in\[−τ,τ\]\[\-\\tau,\\tau\]\. Thus,τ\\taucontrols both the concentration of the stochastic policy and the scale of the policy\-gradient update\. Given actions\{at\}\\\{a\_\{t\}\\\}and advantages\{At\}\\\{A\_\{t\}\\\}, we form𝚲∈ℝT×\|𝒜\|\\boldsymbol\{\\Lambda\}\\in\\mathbb\{R\}^\{T\\times\|\\mathcal\{A\}\|\}and update:

Λt,a\\displaystyle\\Lambda\_\{t,a\}=Atτ\(𝟏\[a=at\]−π\(a∣xt\)\),\\displaystyle=A\_\{t\}\\,\\tau\\\!\\left\(\\mathbf\{1\}\[a=a\_\{t\}\]\-\\pi\(a\\mid x\_\{t\}\)\\right\),\(7\)𝐂\\displaystyle\\mathbf\{C\}←row​\-​norm⁡\(𝐂\+η​𝚲⊤​𝐒\)\.\\displaystyle\\leftarrow\\operatorname\{row\\text\{\-\}norm\}\\\!\\left\(\\mathbf\{C\}\+\\eta\\,\\boldsymbol\{\\Lambda\}^\{\\top\}\\mathbf\{S\}\\right\)\.\(8\)whereη\>0\\eta\>0is the learning rate,𝐆=𝚲⊤​𝐒\\mathbf\{G\}=\\boldsymbol\{\\Lambda\}^\{\\top\}\\mathbf\{S\}is one matrix multiplication, and row normalization restores unit norms\. The update is closed\-form: no gradient is backpropagated through the encoder or a neural actor, and no optimizer state is kept\. Since the per\-row step scales withη​τ\\eta\\tauwhileτ\\taualso sets the logit scale \([6](https://arxiv.org/html/2608.18404#S4.E6)\), the two should therefore be selected jointly\. VSPG is compatible with any advantage estimator\. When Monte\-Carlo estimates are unreliable due to sparse rewards, a critic may optionally be used during training to compute advantages, as in actor–critic methods\[[14](https://arxiv.org/html/2608.18404#bib.bib46)\]\. The critic does not update the action hypervectors or the encoder and is discarded at deployment\.

#### 4\.1\.1Encoders

Every encoder is built from a fixed random base mapφ\\varphi, drawn once and never trained:

φRFF​\(x\)\\displaystyle\\varphi\_\{\\mathrm\{RFF\}\}\(x\)=2D​cos⁡\(𝐖RFF​x\+𝐛\)\\displaystyle=\\sqrt\{\\frac\{2\}\{D\}\}\\,\\cos\(\\mathbf\{W\}\_\{\\mathrm\{RFF\}\}x\+\\mathbf\{b\}\)\(RFF\),\\displaystyle\\text\{\(RFF\)\},\(9\)φFHRR​\(x\)\\displaystyle\\varphi\_\{\\mathrm\{FHRR\}\}\(x\)=2D​\[cos⁡\(𝐖FHRR​x\)sin⁡\(𝐖FHRR​x\)\]\\displaystyle=\\sqrt\{\\frac\{2\}\{D\}\}\\begin\{bmatrix\}\\cos\(\\mathbf\{W\}\_\{\\mathrm\{FHRR\}\}x\)\\\\ \\sin\(\\mathbf\{W\}\_\{\\mathrm\{FHRR\}\}x\)\\end\{bmatrix\}\(FHRR\),\\displaystyle\\text\{\(FHRR\)\},φBasis​\(x\)\\displaystyle\\varphi\_\{\\mathrm\{Basis\}\}\(x\)=ρ⁡\(𝐖Basis​x\),ρ∈\{id,sign\}\\displaystyle=\\rho\\\!\\left\(\\mathbf\{W\}\_\{\\mathrm\{Basis\}\}x\\right\),\\qquad\\rho\\in\\\{\\operatorname\{id\},\\operatorname\{sign\}\\\}\(Basis\)\.\\displaystyle\\text\{\(Basis\)\}\.Here,𝐖RFF,𝐖Basis∈ℝD×d\\mathbf\{W\}\_\{\\mathrm\{RFF\}\},\\mathbf\{W\}\_\{\\mathrm\{Basis\}\}\\in\\mathbb\{R\}^\{D\\times d\}and𝐖FHRR∈ℝ\(D/2\)×d\\mathbf\{W\}\_\{\\mathrm\{FHRR\}\}\\in\\mathbb\{R\}^\{\(D/2\)\\times d\}\. The entries of𝐖RFF\\mathbf\{W\}\_\{\\mathrm\{RFF\}\}and𝐖FHRR\\mathbf\{W\}\_\{\\mathrm\{FHRR\}\}are drawn independently from𝒩⁡\(0,σ−2\)\\mathcal\{N\}\(0,\\sigma^\{\-2\}\), those of𝐖Basis\\mathbf\{W\}\_\{\\mathrm\{Basis\}\}from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\), andbi∼𝒰⁡\(0,2​π\)b\_\{i\}\\sim\\mathcal\{U\}\(0,2\\pi\)\. We defineid⁡\(z\)=z\\operatorname\{id\}\(z\)=z\. The raw encodingϕ~​\(x\)\\widetilde\{\\boldsymbol\{\\phi\}\}\(x\)is eitherφ⁡\(x\)\\varphi\(x\)applied directly or a superposition of boundφ\\varphi\-encodings, followed by \([4](https://arxiv.org/html/2608.18404#S4.E4)\)\. The FHRR map representsei​𝐖​xe^\{i\\mathbf\{W\}x\}through its real and imaginary parts, giving unit norm andφFHRR​\(x\)⊤​φFHRR​\(y\)=2D​∑j=1D/2cos⁡\(𝐰j⊤​\(x−y\)\)\.\\varphi\_\{\\mathrm\{FHRR\}\}\(x\)^\{\\top\}\\varphi\_\{\\mathrm\{FHRR\}\}\(y\)=\\frac\{2\}\{D\}\\sum\_\{j=1\}^\{D/2\}\\cos\\\!\\bigl\(\\mathbf\{w\}\_\{j\}^\{\\top\}\(x\-y\)\\bigr\)\.After normalization, the Basis map approximates cosine similarity whenρ=id\\rho=\\operatorname\{id\}and the angular kernel1−2​θ​\(x,y\)/π1\-2\\theta\(x,y\)/\\piwhenρ=sign\\rho=\\operatorname\{sign\}, whereθ⁡\(x,y\)=arccos⁡\(x⊤​y‖x‖2​‖y‖2\)\.\\theta\(x,y\)=\\arccos\\\!\\left\(\\frac\{x^\{\\top\}y\}\{\\\|x\\\|\_\{2\}\\\|y\\\|\_\{2\}\}\\right\)\.

Normalized inner products then concentrate around an encoder\-specific similarity withκ⁡\(x,x\)=1\\kappa\(x,x\)=1, tightening asDDgrows\[[27](https://arxiv.org/html/2608.18404#bib.bib24),[26](https://arxiv.org/html/2608.18404#bib.bib7),[33](https://arxiv.org/html/2608.18404#bib.bib47)\]: when𝔼⁡\[ϕ~​\(x\)⊤​ϕ~​\(y\)\]=k⁡\(x,y\)\\mathbb\{E\}\\bigl\[\\tilde\{\\boldsymbol\{\\phi\}\}\(x\)^\{\\\!\\top\}\\tilde\{\\boldsymbol\{\\phi\}\}\(y\)\\bigr\]=k\(x,y\), the normalization identifies

ϕ​\(x\)⊤​ϕ​\(y\)≈k⁡\(x,y\)k⁡\(x,x\)​k​\(y,y\)=κ⁡\(x,y\)\.\\boldsymbol\{\\phi\}\(x\)^\{\\\!\\top\}\\boldsymbol\{\\phi\}\(y\)\\;\\approx\\;\\frac\{k\(x,y\)\}\{\\sqrt\{k\(x,x\)\\,k\(y,y\)\}\}\\;=\\;\\kappa\(x,y\)\.\(10\)The analysis below uses onlyϕ​\(xt\)⊤​ϕ​\(x\)≈κ⁡\(xt,x\)\\boldsymbol\{\\phi\}\(x\_\{t\}\)^\{\\\!\\top\}\\boldsymbol\{\\phi\}\(x\)\\approx\\kappa\(x\_\{t\},x\); the encoder determinesκ\\kappa, and hence how advantage evidence generalizes across inputs, while the VSA dimensionalityDDcontrols how accurately the kernel is approximated, which the dimensionality ablation probes directly\.

##### Computational and memory cost\.

Encoding costs one matrix–vector product,O⁡\(D​d\)O\(Dd\)for input dimensiondd, scoring costsO⁡\(\|𝒜\|​D\)O\(\|\\mathcal\{A\}\|D\), and one update costsO⁡\(T​\|𝒜\|​D\)O\(T\|\\mathcal\{A\}\|D\)for𝚲⊤​𝐒\\boldsymbol\{\\Lambda\}^\{\\top\}\\mathbf\{S\}plusO⁡\(T​\|𝒜\|\)O\(T\|\\mathcal\{A\}\|\)for the softmax terms, with no optimizer state\. Because𝐖\\mathbf\{W\}and𝐛\\mathbf\{b\}are random and never trained, deployment stores the\|𝒜\|×D\|\\mathcal\{A\}\|\\times Daction memories together with either the projection or the seed that regenerates it, and the bipolar variant stores one bit per memory coordinate\.

### 4\.2Theoretical Analysis of VSPG

VSPG’s actor design raises two questions: what is the policy\-gradient update for the policy \([5](https://arxiv.org/html/2608.18404#S4.E5)\), and what do the trained action hypervectors store? Proposition[1](https://arxiv.org/html/2608.18404#Thmproposition1)answers the first — the closed\-form rule \([8](https://arxiv.org/html/2608.18404#S4.E8)\)*is*the softmax policy\-gradient step, written as advantage\-weighted bundling followed by row\-wise sphere projection — and Proposition[2](https://arxiv.org/html/2608.18404#Thmproposition2)the second: each trained hypervector is a compressed kernel expansion over experience\. Throughout, let\{\(xt,at,At\)\}t=1T\\\{\(x\_\{t\},a\_\{t\},A\_\{t\}\)\\\}\_\{t=1\}^\{T\}be a batch of transitions and define the empirical surrogate

J^​\(𝐂\)=∑t=1TAt​log⁡π𝐂​\(at∣xt\),\\hat\{J\}\(\\mathbf\{C\}\)\\;=\\;\\sum\_\{t=1\}^\{T\}A\_\{t\}\\log\\pi\_\{\\mathbf\{C\}\}\(a\_\{t\}\\mid x\_\{t\}\),\(11\)where\{At\}\\\{A\_\{t\}\\\}are treated as fixed weights when differentiating with respect to𝐂\\mathbf\{C\}\. We first state the projection fact\.

###### Lemma 2\(Closest unit vector\)\.

For any𝐯≠𝟎\\mathbf\{v\}\\neq\\mathbf\{0\}, the unique unit vector closest to𝐯\\mathbf\{v\}in Euclidean distance is𝐯/‖𝐯‖2\\mathbf\{v\}/\\\|\\mathbf\{v\}\\\|\_\{2\}\.

###### Proof\.

On‖𝐮‖2=1\\\|\\mathbf\{u\}\\\|\_\{2\}=1,‖𝐮−𝐯‖22=1\+‖𝐯‖22−2​𝐮⊤​𝐯\\\|\\mathbf\{u\}\-\\mathbf\{v\}\\\|\_\{2\}^\{2\}=1\+\\\|\\mathbf\{v\}\\\|\_\{2\}^\{2\}\-2\\,\\mathbf\{u\}^\{\\top\}\\mathbf\{v\}, so minimizing distance maximizes𝐮⊤​𝐯\\mathbf\{u\}^\{\\top\}\\mathbf\{v\}, which by Cauchy–Schwarz occurs uniquely at𝐮=𝐯/‖𝐯‖2\\mathbf\{u\}=\\mathbf\{v\}/\\\|\\mathbf\{v\}\\\|\_\{2\}\. ∎

###### Proposition 1\(The VSPG update is a projected policy\-gradient step\)\.

Fix the encoderϕ\\boldsymbol\{\\phi\}and the policy \([5](https://arxiv.org/html/2608.18404#S4.E5)\)\. For any advantages\{At\}\\\{A\_\{t\}\\\},

∇𝐂J^​\(𝐂\)=𝚲⊤​𝐒=𝐆,\\nabla\_\{\\mathbf\{C\}\}\\hat\{J\}\(\\mathbf\{C\}\)\\;=\\;\\boldsymbol\{\\Lambda\}^\{\\top\}\\mathbf\{S\}\\;=\\;\\mathbf\{G\},with𝚲\\boldsymbol\{\\Lambda\}as in \([7](https://arxiv.org/html/2608.18404#S4.E7)\): the bundling term in \([8](https://arxiv.org/html/2608.18404#S4.E8)\) is exactly the sampled policy gradient, and \([8](https://arxiv.org/html/2608.18404#S4.E8)\) is a gradient\-ascent step onJ^\\hat\{J\}followed by row\-wise projection onto the unit sphere \(Lemma[2](https://arxiv.org/html/2608.18404#Thmlemma2)\)\.

###### Proof\.

Writelog⁡π⁡\(at∣xt\)=ℓat​\(xt\)−log⁡Zt\\log\\pi\(a\_\{t\}\\mid x\_\{t\}\)=\\ell\_\{a\_\{t\}\}\(x\_\{t\}\)\-\\log Z\_\{t\}withZt=∑beℓb​\(xt\)Z\_\{t\}=\\sum\_\{b\}e^\{\\ell\_\{b\}\(x\_\{t\}\)\}\. A logitℓb\\ell\_\{b\}depends on row𝐜a\\mathbf\{c\}\_\{a\}only whenb=ab=a, with∇𝐜aℓa=τ​𝐬t\\nabla\_\{\\mathbf\{c\}\_\{a\}\}\\ell\_\{a\}=\\tau\\,\\mathbf\{s\}\_\{t\}, and the log\-sum\-exp derivative gives∇𝐜a​log​Zt=π⁡\(a∣xt\)​τ​𝐬t\\nabla\_\{\\mathbf\{c\}\_\{a\}\}\\log Z\_\{t\}=\\pi\(a\\mid x\_\{t\}\)\\,\\tau\\,\\mathbf\{s\}\_\{t\}\. Subtracting yields the softmax score

∇𝐜alogπ\(at∣xt\)=τ\(𝟏\[a=at\]−π\(a∣xt\)\)𝐬t:\\nabla\_\{\\mathbf\{c\}\_\{a\}\}\\log\\pi\(a\_\{t\}\\mid x\_\{t\}\)=\\tau\\\!\\left\(\\mathbf\{1\}\[a=a\_\{t\}\]\-\\pi\(a\\mid x\_\{t\}\)\\right\)\\mathbf\{s\}\_\{t\}:\(12\)the encoded state is added to the taken action’s row and subtracted from every row in proportion to its current probability\. Weighting each score byAtA\_\{t\}and summing over the batch matches𝚲\\boldsymbol\{\\Lambda\}entrywise, so∇𝐂J^=𝚲⊤​𝐒=𝐆\\nabla\_\{\\mathbf\{C\}\}\\hat\{J\}=\\boldsymbol\{\\Lambda\}^\{\\top\}\\mathbf\{S\}=\\mathbf\{G\}\. Row normalization rescales each row to unit length, which by Lemma[2](https://arxiv.org/html/2608.18404#Thmlemma2)projects𝐂\+η​𝐆\\mathbf\{C\}\+\\eta\\mathbf\{G\}onto the product of unit spheresℳ=\{𝐂:‖𝐜a‖2=1​∀a\}\\mathcal\{M\}=\\\{\\mathbf\{C\}:\\\|\\mathbf\{c\}\_\{a\}\\\|\_\{2\}=1\\;\\forall a\\\}\. ∎

Proposition[1](https://arxiv.org/html/2608.18404#Thmproposition1)gives the exact projected policy\-gradient update\. The following relates this projection to the intrinsic geometry of the unit sphere\.

###### Corollary 1\(First\-order form of the normalized step\)\.

Let𝐠a=∇𝐜aJ^​\(𝐂\)\\mathbf\{g\}\_\{a\}=\\nabla\_\{\\mathbf\{c\}\_\{a\}\}\\hat\{J\}\(\\mathbf\{C\}\), and let𝐜a\+\\mathbf\{c\}\_\{a\}^\{\+\}denote rowaaafter \([8](https://arxiv.org/html/2608.18404#S4.E8)\)\. Ifη​‖𝐠a‖2≤1/4\\eta\\\|\\mathbf\{g\}\_\{a\}\\\|\_\{2\}\\leq 1/4, then

‖𝐜a\+−𝐜a−η⁡\(𝐈−𝐜a​𝐜a⊤\)​𝐠a‖2≤3​η2​‖𝐠a‖22\.\\left\\\|\\mathbf\{c\}\_\{a\}^\{\+\}\-\\mathbf\{c\}\_\{a\}\-\\eta\(\\mathbf\{I\}\-\\mathbf\{c\}\_\{a\}\\mathbf\{c\}\_\{a\}^\{\\top\}\)\\mathbf\{g\}\_\{a\}\\right\\\|\_\{2\}\\leq 3\\eta^\{2\}\\\|\\mathbf\{g\}\_\{a\}\\\|\_\{2\}^\{2\}\.\(13\)Thus, the normalized update is first\-order equivalent to Riemannian gradient ascent on the product of unit spheres\. Moreover,‖𝐠a‖2≤τ​∑t\|At\|\\\|\\mathbf\{g\}\_\{a\}\\\|\_\{2\}\\leq\\tau\\sum\_\{t\}\|A\_\{t\}\|, so the condition holds wheneverη​τ​∑t\|At\|≤1/4\\eta\\tau\\sum\_\{t\}\|A\_\{t\}\|\\leq 1/4\.

###### Proof\.

Write𝐠a=s​𝐜a\+𝐡\\mathbf\{g\}\_\{a\}=s\\mathbf\{c\}\_\{a\}\+\\mathbf\{h\}, wheres=𝐜a⊤​𝐠as=\\mathbf\{c\}\_\{a\}^\{\\top\}\\mathbf\{g\}\_\{a\}and𝐡=\(𝐈−𝐜a​𝐜a⊤\)​𝐠a⟂𝐜a\\mathbf\{h\}=\(\\mathbf\{I\}\-\\mathbf\{c\}\_\{a\}\\mathbf\{c\}\_\{a\}^\{\\top\}\)\\mathbf\{g\}\_\{a\}\\perp\\mathbf\{c\}\_\{a\}\. WithN=‖𝐜a\+η​𝐠a‖2N=\\\|\\mathbf\{c\}\_\{a\}\+\\eta\\mathbf\{g\}\_\{a\}\\\|\_\{2\},𝐜a\+=\(1\+η​s\)​𝐜a\+η​𝐡N\.\\mathbf\{c\}\_\{a\}^\{\+\}=\\frac\{\(1\+\\eta s\)\\mathbf\{c\}\_\{a\}\+\\eta\\mathbf\{h\}\}\{N\}\.Letu=η​‖𝐠a‖2≤1/4u=\\eta\\\|\\mathbf\{g\}\_\{a\}\\\|\_\{2\}\\leq 1/4\. ThenN≥1−u≥3/4N\\geq 1\-u\\geq 3/4and\|1\+η​sN−1\|≤u2,\|1N−1\|≤u1−u≤2​u\.\\left\|\\frac\{1\+\\eta s\}\{N\}\-1\\right\|\\leq u^\{2\},\\qquad\\left\|\\frac\{1\}\{N\}\-1\\right\|\\leq\\frac\{u\}\{1\-u\}\\leq 2u\.Since the radial and tangential terms are orthogonal,‖𝐜a\+−𝐜a−η​𝐡‖2≤u4\+4​u4<3​u2\.\\left\\\|\\mathbf\{c\}\_\{a\}^\{\+\}\-\\mathbf\{c\}\_\{a\}\-\\eta\\mathbf\{h\}\\right\\\|\_\{2\}\\leq\\sqrt\{u^\{4\}\+4u^\{4\}\}<3u^\{2\}\.Finally,‖𝐠a‖2≤∑t\|Λt,a\|​‖𝐬t‖2≤τ​∑t\|At\|\\\|\\mathbf\{g\}\_\{a\}\\\|\_\{2\}\\leq\\sum\_\{t\}\|\\Lambda\_\{t,a\}\|\\\|\\mathbf\{s\}\_\{t\}\\\|\_\{2\}\\leq\\tau\\sum\_\{t\}\|A\_\{t\}\|\. ∎

![Refer to caption](https://arxiv.org/html/2608.18404v1/result01.png)Figure 2:Overall performance analysis of VSPG compared with different baselines and encoders runs on 5 seeds\. Except forLunaLander\-v2, VSPG\-based achieves the fastest convergence and competitive performance compared to baselines\.Proposition[1](https://arxiv.org/html/2608.18404#Thmproposition1)places VSPG within standard policy\-gradient theory: the actor is a log\-linear policy over fixed hypervector features\[[32](https://arxiv.org/html/2608.18404#bib.bib10),[21](https://arxiv.org/html/2608.18404#bib.bib31)\], and under the unit\-norm constraint its exact gradient step takes the form of an VSA bundling operation\. Alternative advantage estimates or surrogate weights change only the entries of𝚲\\boldsymbol\{\\Lambda\}\.

###### Proposition 2\(Exact expansion of trained action hypervectors\)\.

Run Algorithm[1](https://arxiv.org/html/2608.18404#alg1)for any number of updates from unit\-norm initialization\{𝐜a\(0\)\}\\\{\\mathbf\{c\}\_\{a\}^\{\(0\)\}\\\}, and let\{xk\}k=1N\\\{x\_\{k\}\\\}\_\{k=1\}^\{N\}collect all inputs visited during training\. Then there exist scalarsβa\>0\\beta\_\{a\}\>0and\{αk,a\}\\\{\\alpha\_\{k,a\}\\\}such that each trained action hypervector is

𝐜a=βa​𝐜a\(0\)\+∑k=1Nαk,a​ϕ​\(xk\),\\mathbf\{c\}\_\{a\}\\;=\\;\\beta\_\{a\}\\,\\mathbf\{c\}\_\{a\}^\{\(0\)\}\\;\+\\;\\sum\_\{k=1\}^\{N\}\\alpha\_\{k,a\}\\,\\boldsymbol\{\\phi\}\(x\_\{k\}\),\(14\)and consequently the logit at any query inputxxdecomposes as

ℓa​\(x\)=\\displaystyle\\ell\_\{a\}\(x\)=τ​βa​𝐜a\(0\)⊤​ϕ​\(x\)⏟initialization;​O​\(1/D\)\+τ​∑k=1Nαk,a​ϕ​\(xk\)⊤​ϕ​\(x\)⏟≈κ⁡\(xk,x\)\.\\displaystyle\\underbrace\{\\tau\\beta\_\{a\}\\,\\mathbf\{c\}\_\{a\}^\{\(0\)\\top\}\\boldsymbol\{\\phi\}\(x\)\}\_\{\\text\{initialization; \}O\(1/\\sqrt\{D\}\)\}\+\\tau\\\!\\sum\_\{k=1\}^\{N\}\\alpha\_\{k,a\}\\,\\underbrace\{\\boldsymbol\{\\phi\}\(x\_\{k\}\)^\{\\\!\\top\}\\boldsymbol\{\\phi\}\(x\)\}\_\{\\approx\\,\\kappa\(x\_\{k\},\\,x\)\}\.\(15\)
Writingza\(j\)\>0z\_\{a\}^\{\(j\)\}\>0for the Euclidean norm of rowaaafter thejj\-th bundling step and before its renormalization, andΛ\(j\)\\Lambda^\{\(j\)\},\{xt\(j\)\}\\\{x\_\{t\}^\{\(j\)\}\\\}for the weights and inputs of thejj\-th batch, the coefficients are

βa=∏j=1J\(za\(j\)\)−1,αk,a=η∑\(j,t\):xt\(j\)=xkΛt,a\(j\)∏i=jJ\(za\(i\)\)−1\.\\beta\_\{a\}=\\prod\_\{j=1\}^\{J\}\\bigl\(z\_\{a\}^\{\(j\)\}\\bigr\)^\{\-1\},\\alpha\_\{k,a\}=\\eta\\\!\\\!\\sum\_\{\(j,t\)\\,:\\,x\_\{t\}^\{\(j\)\}=x\_\{k\}\}\\\!\\\!\\Lambda\_\{t,a\}^\{\(j\)\}\\prod\_\{i=j\}^\{J\}\\bigl\(z\_\{a\}^\{\(i\)\}\\bigr\)^\{\-1\}\.\(16\)Eachαk,a\\alpha\_\{k,a\}is therefore a positively weighted sum of the advantage\-weighted softmax scores collected at the visits ofxkx\_\{k\}; in particular, ifxkx\_\{k\}is visited once at steptt, thensign\(αk,a\)=sign\(At\(𝟏\[a=at\]−π\(a∣xt\)\)\)\\operatorname\{sign\}\(\\alpha\_\{k,a\}\)=\\operatorname\{sign\}\\bigl\(A\_\{t\}\(\\mathbf\{1\}\[a=a\_\{t\}\]\-\\pi\(a\\mid x\_\{t\}\)\)\\bigr\)\.

###### Proof\.

By induction\. At initialization \([14](https://arxiv.org/html/2608.18404#S4.E14)\) holds withβa=1\\beta\_\{a\}=1,αk,a=0\\alpha\_\{k,a\}=0\. Each update \([8](https://arxiv.org/html/2608.18404#S4.E8)\) addsη​∑tΛt,a​𝐬t\\eta\\sum\_\{t\}\\Lambda\_\{t,a\}\\,\\mathbf\{s\}\_\{t\}, which lies in the span of encoded visited inputs, and row normalization rescales the row by a positive scalar, preserving the form of \([14](https://arxiv.org/html/2608.18404#S4.E14)\) and the coefficient signs\. Unrolling the recursion𝐜a\(j\)=\(𝐜a\(j−1\)\+η​∑tΛt,a\(j\)​𝐬t\(j\)\)/za\(j\)\\mathbf\{c\}\_\{a\}^\{\(j\)\}=\\bigl\(\\mathbf\{c\}\_\{a\}^\{\(j\-1\)\}\+\\eta\\sum\_\{t\}\\Lambda\_\{t,a\}^\{\(j\)\}\\mathbf\{s\}\_\{t\}^\{\(j\)\}\\bigr\)/z\_\{a\}^\{\(j\)\}yields the coefficients in \([16](https://arxiv.org/html/2608.18404#S4.E16)\), whose weights are positive because everyza\(i\)z\_\{a\}^\{\(i\)\}is positive\. Inner products of \([14](https://arxiv.org/html/2608.18404#S4.E14)\) withτ​ϕ​\(x\)\\tau\\boldsymbol\{\\phi\}\(x\)give \([15](https://arxiv.org/html/2608.18404#S4.E15)\)\. ∎

Informally, up to a vanishing initialization bias, the deployed VSPG policy is a softmax over*advantage\-weighted kernel scores against experience*:

π⁡\(a∣x\)≈softmaxa⁡\(τ​∑k=1Nαk,a​κ​\(xk,x\)\)\.\\pi\(a\\mid x\)\\;\\approx\\;\\operatorname\{softmax\}\_\{a\}\\\!\\Bigl\(\\tau\\sum\_\{k=1\}^\{N\}\\alpha\_\{k,a\}\\,\\kappa\(x\_\{k\},x\)\\Bigr\)\.\(17\)
An advantageous transition contributes positive mass to actionaaat encoder\-similar inputs, while evidence favoring competing actions contributes negative mass\. This kernel sharing allows VSPG to reuse each transition across a neighborhood of observations, providing a representation\-level mechanism for sample\-efficient learning\. Generalization is therefore governed by the encoder\-induced similarity, while theNN\-term expansion remains superposed inDDfixed coordinates and is never enumerated at inference\.

The same update applies independently to each agent in multi\-agent training, using agent\-specific advantages that may be estimated by a centralized critic during training\. At deployment, only the fixed encoder and action memories are retained, and actions are selected bya∗=arg⁡maxa​𝐜a⊤​ϕ​\(x\)\.a^\{\*\}=\\arg\\max\_\{a\}\\mathbf\{c\}\_\{a\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)\.By \([17](https://arxiv.org/html/2608.18404#S4.E17)\), this fixed\-size readout evaluates the compressed kernel memory without storing or enumerating the training samples\.

### 4\.3Robustness of the Stored Policy

For bipolar action memories, independent sign flips admit a direct stability guarantee for the greedy readout\.

###### Proposition 3\(Bit\-flip stability of the greedy readout\)\.

Let the deployed actor store bipolar action memories𝐜a∈\{−1/D,\+1/D\}D\\mathbf\{c\}\_\{a\}\\in\\\{\-1/\\sqrt\{D\},\+1/\\sqrt\{D\}\\\}^\{D\}and select actions bya∗​\(x\)=arg⁡maxa​𝐜a⊤​ϕ​\(x\)a^\{\*\}\(x\)=\\arg\\max\_\{a\}\\mathbf\{c\}\_\{a\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)with‖ϕ⁡\(x\)‖2=1\\\|\\boldsymbol\{\\phi\}\(x\)\\\|\_\{2\}=1\. Suppose each stored coordinate flips sign independently with probabilityp<1/2p<1/2, giving corrupted memories𝐜~a\\tilde\{\\mathbf\{c\}\}\_\{a\}\. Fix an inputxxand letΔ⁡\(x\)=𝐜a∗⊤​ϕ​\(x\)−maxb≠a∗⁡𝐜b⊤​ϕ​\(x\)\>0\\Delta\(x\)=\\mathbf\{c\}\_\{a^\{\*\}\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)\-\\max\_\{b\\neq a^\{\*\}\}\\mathbf\{c\}\_\{b\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)\>0denote the similarity margin\. Then

Pr\[argmaxa𝐜~a⊤ϕ\(x\)≠a∗\(x\)\]≤2\|𝒜\|exp\(−D​\(1−2​p\)2​Δ​\(x\)28\)\.\\Pr\\Bigl\[\\arg\\max\_\{a\}\\tilde\{\\mathbf\{c\}\}\_\{a\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)\\neq a^\{\*\}\(x\)\\Bigr\]\\leq 2\\,\|\\mathcal\{A\}\|\\,\\exp\\\!\\Bigl\(\-\\tfrac\{D\(1\-2p\)^\{2\}\\Delta\(x\)^\{2\}\}\{8\}\\Bigr\)\.\(18\)

###### Proof\.

Writec~a,j=σa,j​ca,j\\tilde\{c\}\_\{a,j\}=\\sigma\_\{a,j\}\\,c\_\{a,j\}with independentσa,j∈\{−1,\+1\}\\sigma\_\{a,j\}\\in\\\{\-1,\+1\\\}andPr\[σa,j=−1\]=p\\Pr\[\\sigma\_\{a,j\}=\-1\]=p, so𝔼⁡\[σa,j\]=1−2​p\\mathbb\{E\}\[\\sigma\_\{a,j\}\]=1\-2p\. Then𝐜~a⊤​ϕ​\(x\)=\(1−2​p\)​𝐜a⊤​ϕ​\(x\)\+ζa\\tilde\{\\mathbf\{c\}\}\_\{a\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)=\(1\-2p\)\\,\\mathbf\{c\}\_\{a\}^\{\\top\}\\boldsymbol\{\\phi\}\(x\)\+\\zeta\_\{a\}withζa=∑j\(σa,j−\(1−2​p\)\)​ca,j​ϕj​\(x\)\\zeta\_\{a\}=\\sum\_\{j\}\\bigl\(\\sigma\_\{a,j\}\-\(1\-2p\)\\bigr\)c\_\{a,j\}\\phi\_\{j\}\(x\), a sum of independent zero\-mean terms whosejj\-th term has range2​\|ca,j​ϕj​\(x\)\|2\|c\_\{a,j\}\\phi\_\{j\}\(x\)\|\. Since∑j\(2​\|ca,j​ϕj​\(x\)\|\)2=4D​∑jϕj​\(x\)2=4D\\sum\_\{j\}\\bigl\(2\|c\_\{a,j\}\\phi\_\{j\}\(x\)\|\\bigr\)^\{2\}=\\tfrac\{4\}\{D\}\\sum\_\{j\}\\phi\_\{j\}\(x\)^\{2\}=\\tfrac\{4\}\{D\}, Hoeffding’s inequality givesPr\[\|ζa\|≥t\]≤2exp\(−Dt2/2\)\\Pr\[\|\\zeta\_\{a\}\|\\geq t\]\\leq 2\\exp\(\-Dt^\{2\}/2\)for everyaa\. Scaling all similarities by1−2​p\>01\-2p\>0preserves the maximizer, so the corrupted argmax can change only if\|ζa\|≥\(1−2​p\)​Δ​\(x\)/2\|\\zeta\_\{a\}\|\\geq\(1\-2p\)\\Delta\(x\)/2for someaa\. A union bound over the\|𝒜\|\|\\mathcal\{A\}\|actions completes the proof\. ∎

## 5Experiments

### 5\.1Experimental Setup and Implementation Details

To evaluate VSPG and the empirical implications of our theory, we use MiniGrid\[[6](https://arxiv.org/html/2608.18404#bib.bib38)\], classic control, and SustainGym building control\[[38](https://arxiv.org/html/2608.18404#bib.bib40)\]\. The first two test sample efficiency and final performance, while SustainGym tests continuous, noisy, delayed, and multi\-agent control\. Tuning budgets are comparable and otherwise favor the baselines; full details are in the supplementary material\. Experiments use either one RTX 4090 or CPUs, and all results are reproducible on CPUs alone\. Unless otherwise noted, all methods use discount factorγ=0\.99\\gamma=0\.99, and VSPG uses dimensionalityD=10,000D=10\{,\}000for MiniGrid and classic control andD=5,000D=5\{,\}000for SustainGym\.

### 5\.2MiniGrid and Classic Control

We first evaluate VSPG on standard single\-agent discrete\-action benchmarks: classic\-control environments from Gymnasium\[[34](https://arxiv.org/html/2608.18404#bib.bib37)\]and navigation tasks from MiniGrid\[[6](https://arxiv.org/html/2608.18404#bib.bib38)\]\. These experiments assess sample efficiency, final performance, and comparison with policy\-gradient and prior VSA\-based RL baselines\. Following QHD\[[23](https://arxiv.org/html/2608.18404#bib.bib2)\], the classic\-control suite includesCartPole\-v1,LunarLander\-v2, andAcrobot\-v1, providing a direct comparison with prior VSA reinforcement learning on low\-dimensional continuous\-observation tasks\. We additionally evaluateEmpty\-5x5,DoorKey\-5x5, andDoorKey\-8x8\. These MiniGrid tasks introduce partial observability, sparse rewards, and longer\-horizon credit assignment:Empty\-5x5primarily tests rapid goal reaching, whereas theDoorKeytasks require coordinated key pickup, door unlocking, and navigation to the goal\. Exact encoder constructions, observation preprocessing, and task\-specific configurations are provided in the supplementary material\.

![Refer to caption](https://arxiv.org/html/2608.18404v1/bitflip2.png)Figure 3:Bit\-flip robustness on SustainGym after quantization\. VSPG, DNN, andRaw\-Linear actors are evaluated\.
![Refer to caption](https://arxiv.org/html/2608.18404v1/dim.png)Figure 4:Dimensionality analysis onCartPole\-v1andDoorKey 8x8\.

![Refer to caption](https://arxiv.org/html/2608.18404v1/prop2_encoder_ablation.png)Figure 5:Encoder\-induced kernel neighborhoods from trained VSPG trajectories onDoorKey\-8×88\\times 8\.We compare VSPG against three baselines\.Raw\-Lineartests whether the gains come from VSA encoding rather than from the linear action\-hypervector policy alone\. TheDNNbaseline is a two\-hidden\-layer neural policy trained with backpropagation under the same advantage\-estimation protocol where applicable, allowing us to compare sample efficiency against a standard neural policy\. We also includeQHDas the closest prior VSA baseline for single\-agent discrete\-action RL\. Since QHD is value\-based and off\-policy, whereas VSPG is policy\-based and on\-policy, we match the total episode budget rather than the number of parameter updates\. MiniGrid experiments use2,0002\{,\}000episodes for5×55\\times 5grids and5,0005\{,\}000episodes for the8×88\\times 8grid, while classic\-control experiments use10,00010\{,\}000episodes\. All VSPG and policy\-gradient baselines are trained with REINFORCE except forAcrobot\-v1, where all policy\-gradient methods use GAE with the same PPO\-style clipped importance weighting\[[30](https://arxiv.org/html/2608.18404#bib.bib45)\], since sparse rewards often keep vanilla REINFORCE near the minimum return of−500\-500\. We report the mean and standard error over five seeds, using the mean reward over the final 100 episodes of each run as the main summary metric\.

Results\.[Figure 2](https://arxiv.org/html/2608.18404#S4.F2)shows that VSPG learns substantially faster than the DNN and linear actors on classic control, especially onCartPole\-v1andAcrobot\-v1\. OnLunarLander\-v2, VSPG improves faster early in training, while all methods reach broadly comparable final performance\. VSPG also remains competitive across MiniGrid, including the more difficult and variableDoorKey\-8x8\. QHD is competitive on simpler classic\-control tasks, but deteriorates as task complexity and partial observability increase: it becomes unstable and collapses onDoorKey\-5x5, and learns almost nothing onDoorKey\-8x8\. Overall, these results are consistent with the kernel\-sharing mechanism in Proposition[2](https://arxiv.org/html/2608.18404#Thmproposition2), through which advantage evidence is reused across encoder\-similar observations, supporting sample\-efficient learning in both fully and partially observable tasks without a neural actor\.

### 5\.3Building Control and Bit\-flip Robustness

Table 2:Average reward per step \(mean±\\pmstd\) on SustainGym building control after 500 training episodes\.We evaluate VSPG on SustainGym\[[38](https://arxiv.org/html/2608.18404#bib.bib40)\], a multi\-agent building\-control benchmark with noisy, non\-stationary physical observations\. We compare against DNN and Raw\-Linear actors trained under the same decentralized\-actor, centralized\-critic multi\-agent pipeline\. The centralized critic is used only during training to compute GAE advantages and is discarded at deployment\. QHD is omitted because it is a single\-agent, value\-based method outside our actor\-focused multi\-agent comparison\. All checkpoints are trained for 500 episodes before evaluation\. Following\[[43](https://arxiv.org/html/2608.18404#bib.bib21),[47](https://arxiv.org/html/2608.18404#bib.bib11)\], we post\-training quantize each actor’s parameters—the action\-hypervector matrix𝐂\\mathbf\{C\}for VSPG, and all actor weights and biases for DNN and Raw\-Linear—tob∈\{1,2,4,8\}b\\in\\\{1,2,4,8\\\}\-bit signed integers using per\-tensor min–max affine quantization\. We then flip each stored bit independently with probabilityppand dequantize the corrupted parameters back to float32 before evaluation\.

#### 5\.3\.1Results

[Table 2](https://arxiv.org/html/2608.18404#S5.T2)shows that FHRR\- and RFF\-VSPG achieve the strongest returns across both climates, whereas Basis\-VSPG performs poorly and exhibits high variance\. Together with the failure of RFF\-VSPG onDoorKey\-8x8, this indicates that no encoder is uniformly effective and that VSPG inherits the inductive bias of its encoder\-induced similarity\. Under post\-training corruption, however,[Figure 4](https://arxiv.org/html/2608.18404#S5.F4)shows that VSPG action memories generally degrade more gracefully than DNN and Raw\-Linear actors\. Proposition[3](https://arxiv.org/html/2608.18404#Thmproposition3)complements these results for genuinely bipolar memories under direct sign flips, while the affine\-quantized real\-valued memories are evaluated empirically\.

### 5\.4Ablation Studies

We examine two representation\-level implications of Proposition[2](https://arxiv.org/html/2608.18404#Thmproposition2): dimensionality controls how faithfully the kernel expansion is compressed, while the encoder determines the neighborhood over which advantage evidence is shared\.

#### 5\.4\.1Dimensionality and policy memory\.

[Figure 4](https://arxiv.org/html/2608.18404#S5.F4)reports the normalized area under the learning curve from 100\-episode\-smoothed returns or success rates\. OnCartPole\-v1, performance improves withDDand largely saturates beyondD=1,000D=1\{,\}000\.DoorKey\-8x8is more sensitive, but higher dimensions generally improve learning despite variation across seeds\. This trend is consistent with \([10](https://arxiv.org/html/2608.18404#S4.E10)\): increasingDDimproves the approximation ofκ⁡\(x,x′\)\\kappa\(x,x^\{\\prime\}\)and reduces interference in the compressed policy memory\.

#### 5\.4\.2Encoder\-Induced Kernel Neighborhoods\.

For trainedDoorKey\-8×\\times8policies, we retrieve the top\-3 neighbors of sampled observations under the fixed encoder\-induced similarity \([Figure 5](https://arxiv.org/html/2608.18404#S5.F5)\)\. Basis and FHRR produce coherent neighborhoods with high kernel similarity, whereas the unsuccessful RFF configuration yields weaker and less consistent matches\. Together with[Figure 2](https://arxiv.org/html/2608.18404#S4.F2), these ablations illustrate the mechanism in Proposition[2](https://arxiv.org/html/2608.18404#Thmproposition2):DDgoverns interference in the compressed kernel memory, while the encoder determines whether stored advantage evidence is transferred to observations where it supports useful generalization, accounting for both the strong and failed configurations\.

## 6Conclusion

In this paper, we introduced VSPG, a vector\-symbolic formulation of discrete\-action policy gradients\. Its softmax update becomes advantage hypervector bundling, while action memories form fixed\-size kernel expansions over experience\. For bipolar memories, greedy action selection is provably robust to random bit flips, and experiments show favorable sample efficiency, competitive returns, and graceful degradation under quantization and memory corruption\. Future work will test whether these properties extend to vision\-driven decision making and robotics\[[22](https://arxiv.org/html/2608.18404#bib.bib34),[4](https://arxiv.org/html/2608.18404#bib.bib35)\]\.

## References

- \[1\]A\. Agarwal, S\. M\. Kakade, J\. D\. Lee, and G\. Mahajan\(2021\)On the theory of policy gradient methods: optimality, approximation, and distribution shift\.Journal of Machine Learning Research22\(98\),pp\. 1–76\.Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[2\]R\. Arbore, X\. Routh, A\. R\. Noor, A\. Kothari, H\. Yang, W\. Xu, S\. Pinge, M\. Zhou, T\. Rosing, and V\. Adve\(2025\)HPVM\-hdc: a heterogeneous programming system for accelerating hyperdimensional computing\.InProceedings of the 52nd Annual International Symposium on Computer Architecture,ISCA ’25,New York, NY, USA,pp\. 1342–1355\.External Links:ISBN 9798400712616,[Link](https://doi.org/10.1145/3695053.3731095),[Document](https://dx.doi.org/10.1145/3695053.3731095)Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[3\]J\. Arockiaraj, D\. Parikh, and V\. Prasanna\(2026\)ImageHD: energy\-efficient on\-device continual learning of visual representations via hyperdimensional computing\.In2026 IEEE 34th Annual International Symposium on Field\-Programmable Custom Computing Machines \(FCCM\),Vol\.,pp\. 231–240\.External Links:[Document](https://dx.doi.org/10.1109/FCCM68464.2026.00041)Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[4\]M\. Caron, H\. Touvron, I\. Misra, H\. Jégou, J\. Mairal, P\. Bojanowski, and A\. Joulin\(2021\)Emerging properties in self\-supervised vision transformers\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 9650–9660\.Cited by:[§6](https://arxiv.org/html/2608.18404#S6.p1.1)\.
- \[5\]Z\. Chen, D\. Hoang, F\. J\. Piran, R\. Chen, and F\. Imani\(2025\)Federated hyperdimensional computing for hierarchical and distributed quality monitoring in smart manufacturing\.Internet of Things31,pp\. 101568\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[6\]M\. Chevalier\-Boisvert, B\. Dai, M\. Towers, R\. Perez\-Vicente, L\. Willems, S\. Lahlou, S\. Pal, P\. S\. Castro, and J\. K\. Terry\(2023\)Minigrid & miniworld: modular & customizable reinforcement learning environments for goal\-oriented tasks\.Advances in Neural Information Processing Systems36,pp\. 73383–73394\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p5.1),[§5\.1](https://arxiv.org/html/2608.18404#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2608.18404#S5.SS2.p1.1)\.
- \[7\]G\. Dulac\-Arnold, N\. Levine, D\. J\. Mankowitz, J\. Li, C\. Paduraru, S\. Gowal, and T\. Hester\(2021\)Challenges of real\-world reinforcement learning: definitions, benchmarks and analysis\.Machine Learning110\(9\),pp\. 2419–2468\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.18404#S2.SS2.p1.1)\.
- \[8\]A\. N\. Gorban and I\. Y\. Tyukin\(2018\)Blessing of dimensionality: mathematical foundations of the statistical physics of data\.Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences376\(2118\),pp\. 20170237\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[9\]A\. Hernández\-Cano, N\. Matsumoto, E\. Ping, and M\. Imani\(2021\)OnlineHD: robust, efficient, and single\-pass online learning using hyperdimensional system\.In2021 Design, Automation & Test in Europe Conference & Exhibition \(DATE\),Vol\.,pp\. 56–61\.External Links:[Document](https://dx.doi.org/10.23919/DATE51398.2021.9474107)Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.18404#S3.SS2.p3.2),[Remark 1](https://arxiv.org/html/2608.18404#Thmremark1.p1.1)\.
- \[10\]M\. A\. Issa, H\. Chen, J\. Wang, and M\. Imani\(2024\)CyberRL: brain\-inspired reinforcement learning for efficient network intrusion detection\.IEEE Transactions on Computer\-Aided Design of Integrated Circuits and Systems\.Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[11\]S\. M\. Kakade, J\. Schneider, and A\. Ng\(2003\)Policy search by dynamic programming\.Advances in neural information processing systems16\.Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[12\]P\. Kanerva\(2009\)Hyperdimensional computing: an introduction to computing in distributed representation with high\-dimensional random vectors\.Cognitive computation1\(2\),pp\. 139–159\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p1.1),[Remark 1](https://arxiv.org/html/2608.18404#Thmremark1.p1.1)\.
- \[13\]D\. Kleyko, D\. A\. Rachkovskij, E\. Osipov, and A\. Rahimi\(2022\)A survey on hyperdimensional computing aka vector symbolic architectures, part i: models and data transformations\.ACM Comput\. Surv\.55\(6\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3538531),[Document](https://dx.doi.org/10.1145/3538531)Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[14\]V\. Konda and J\. Tsitsiklis\(1999\)Actor\-critic algorithms\.Advances in neural information processing systems12\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.18404#S3.SS1.p1.3),[§4\.1](https://arxiv.org/html/2608.18404#S4.SS1.p1.5)\.
- \[15\]S\. Koppula, L\. Orosa, A\. G\. Yağlıkçı, R\. Azizi, T\. Shahroodi, K\. Kanellopoulos, and O\. Mutlu\(2019\)EDEN: enabling energy\-efficient, high\-performance deep neural network inference using approximate dram\.InProceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture,MICRO\-52,New York, NY, USA,pp\. 166–181\.External Links:ISBN 9781450369381,[Link](https://doi.org/10.1145/3352460.3358280),[Document](https://dx.doi.org/10.1145/3352460.3358280)Cited by:[§2\.2](https://arxiv.org/html/2608.18404#S2.SS2.p1.1)\.
- \[16\]H\. Kwon, K\. Kim, J\. Lee, H\. Lee, J\. Kim, J\. Kim, T\. Kim, Y\. Kim, Y\. Ni, M\. Imani, I\. Suh, and Y\. Kim\(2024\)Brain\-inspired hyperdimensional computing in the wild: lightweight symbolic learning for sensorimotor controls of wheeled robots\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),Vol\.,pp\. 5176–5182\.External Links:[Document](https://dx.doi.org/10.1109/ICRA57147.2024.10610176)Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[17\]C\. Y\. Lee, S\. Achour, and Z\. Kapetanovic\(2025\)NavHD: low\-power learning for micro\-robotic controls in the wild\.In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 13399–13405\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.6.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[18\]H\. Lee, W\. Han, H\. Kim, H\. Kwon, S\. Jang, I\. Suh, and Y\. Kim\(2025\)Hyperdimensional computing\-based federated learning in mobile robots through synthetic oversampling\.In2025 IEEE International Conference on Robotics and Automation \(ICRA\),Vol\.,pp\. 13406–13412\.External Links:[Document](https://dx.doi.org/10.1109/ICRA55743.2025.11127388)Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[19\]G\. Lever and R\. Stafford\(2015\)Modelling Policies in MDPs in Reproducing Kernel Hilbert Space\.InProceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics,G\. Lebanon and S\. V\. N\. Vishwanathan \(Eds\.\),Proceedings of Machine Learning Research, Vol\.38,San Diego, California, USA,pp\. 590–598\.External Links:[Link](https://proceedings.mlr.press/v38/lever15.html)Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.4.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[20\]H\. Li, F\. Liu, Y\. Chen, Z\. Wang, S\. Huang, N\. Yang, D\. Lyu, and L\. Jiang\(2025\)FATE: boosting the performance of hyper\-dimensional computing intelligence with flexible numerical data type\.InProceedings of the 52nd Annual International Symposium on Computer Architecture,ISCA ’25,New York, NY, USA,pp\. 1269–1282\.External Links:ISBN 9798400712616,[Link](https://doi.org/10.1145/3695053.3731031),[Document](https://dx.doi.org/10.1145/3695053.3731031)Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[21\]J\. Mei, C\. Xiao, C\. Szepesvari, and D\. Schuurmans\(2020\)On the global convergence rates of softmax policy gradient methods\.InProceedings of the 37th International Conference on Machine Learning,H\. D\. III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 6820–6829\.External Links:[Link](https://proceedings.mlr.press/v119/mei20b.html)Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.3.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.18404#S4.SS2.p6.1)\.
- \[22\]V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. Riedmiller\(2013\)Playing atari with deep reinforcement learning\.arXiv preprint arXiv:1312\.5602\.Cited by:[§6](https://arxiv.org/html/2608.18404#S6.p1.1)\.
- \[23\]Y\. Ni, D\. Abraham, M\. Issa, Y\. Kim, P\. Mercati, and M\. Imani\(2023\)Efficient off\-policy reinforcement learning via brain\-inspired computing\.InProceedings of the Great Lakes Symposium on VLSI 2023,pp\. 449–453\.Cited by:[Appendix A](https://arxiv.org/html/2608.18404#A1.p3.1),[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.5.1),[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§1](https://arxiv.org/html/2608.18404#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1),[§5\.2](https://arxiv.org/html/2608.18404#S5.SS2.p1.1)\.
- \[24\]Y\. Ni, W\. Y\. Chung, S\. Cho, Z\. Zou, and M\. Imani\(2024\)Efficient exploration in edge\-friendly hyperdimensional reinforcement learning\.InProceedings of the Great Lakes Symposium on VLSI 2024,GLSVLSI ’24,New York, NY, USA,pp\. 111–118\.External Links:ISBN 9798400706059,[Link](https://doi.org/10.1145/3649476.3658760),[Document](https://dx.doi.org/10.1145/3649476.3658760)Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.5.1),[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[25\]Y\. Ni, M\. Issa, D\. Abraham, M\. Imani, X\. Yin, and M\. Imani\(2022\)Hdpg: hyperdimensional policy\-based reinforcement learning for continuous control\.InProceedings of the 59th ACM/IEEE Design Automation Conference,pp\. 1141–1146\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.7.1),[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[26\]T\.A\. Plate\(1995\)Holographic reduced representations\.IEEE Transactions on Neural Networks6\(3\),pp\. 623–641\.External Links:[Document](https://dx.doi.org/10.1109/72.377968)Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p1.1),[§4\.1\.1](https://arxiv.org/html/2608.18404#S4.SS1.SSS1.p2.1)\.
- \[27\]A\. Rahimi and B\. Recht\(2007\)Random features for large\-scale kernel machines\.Advances in neural information processing systems20\.Cited by:[§4\.1\.1](https://arxiv.org/html/2608.18404#S4.SS1.SSS1.p2.1)\.
- \[28\]A\. Rahimi and B\. Recht\(2008\)Weighted sums of random kitchen sinks: replacing minimization with randomization in learning\.Advances in neural information processing systems21\.Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[29\]J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel\(2016\)High\-dimensional continuous control using generalized advantage estimation\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.18404#S3.SS1.p1.3)\.
- \[30\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.2.1),[§3\.1](https://arxiv.org/html/2608.18404#S3.SS1.p1.3),[§5\.2](https://arxiv.org/html/2608.18404#S5.SS2.p2.1)\.
- \[31\]R\. S\. Sutton A\. G\. Bartoet al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§3\.1](https://arxiv.org/html/2608.18404#S3.SS1.p1.3)\.
- \[32\]R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour\(1999\)Policy gradient methods for reinforcement learning with function approximation\.Advances in neural information processing systems12\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.3.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.18404#S3.SS1.p1.3),[§4\.2](https://arxiv.org/html/2608.18404#S4.SS2.p6.1)\.
- \[33\]A\. Thomas, S\. Dasgupta, and T\. Rosing\(2021\)A theoretical perspective on hyperdimensional computing\.Journal of Artificial Intelligence Research72,pp\. 215–249\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1),[§4\.1\.1](https://arxiv.org/html/2608.18404#S4.SS1.SSS1.p2.1)\.
- \[34\]M\. Towers, A\. Kwiatkowski, J\. U\. Balis, G\. D\. Cola, T\. Deleu, M\. Goulão, K\. Andreas, M\. Krimmel, A\. KG, R\. D\. L\. Perez\-Vicente, J\. K\. Terry, A\. Pierré, S\. V\. Schulhoff, J\. J\. Tai, H\. Tan, and O\. G\. Younis\(2026\)Gymnasium: a standard interface for reinforcement learning environments\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=qPMLvJxtPK)Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p5.1),[§5\.2](https://arxiv.org/html/2608.18404#S5.SS2.p1.1)\.
- \[35\]R\. Vershynin\(2018\)High\-dimensional probability: an introduction with applications in data science\.Vol\.47,Cambridge university press\.Cited by:[§3\.2](https://arxiv.org/html/2608.18404#S3.SS2.p2.1.1)\.
- \[36\]Z\. Wan, N\. Chandramoorthy, K\. Swaminathan, P\. Chen, K\. Bhardwaj, V\. J\. Reddi, and A\. Raychowdhury\(2024\)Mulberry: enabling bit\-error robustness for energy\-efficient multi\-agent autonomous systems\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,pp\. 746–762\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.8.1),[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.18404#S2.SS2.p1.1)\.
- \[37\]Z\. Wan, N\. Chandramoorthy, K\. Swaminathan, P\. Chen, V\. J\. Reddi, and A\. Raychowdhury\(2023\)Berry: bit error robustness for energy\-efficient reinforcement learning\-based autonomous systems\.In2023 60th ACM/IEEE Design Automation Conference \(DAC\),pp\. 1–6\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.8.1),[§1](https://arxiv.org/html/2608.18404#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.18404#S2.SS2.p1.1)\.
- \[38\]C\. Yeh, V\. Li, R\. Datta, J\. Arroyo, N\. Christianson, C\. Zhang, Y\. Chen, M\. M\. Hosseini, A\. Golmohammadi, Y\. Shi,et al\.\(2023\)SustainGym: reinforcement learning environments for sustainable energy systems\.Advances in Neural Information Processing Systems36,pp\. 59464–59476\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p5.1),[§5\.1](https://arxiv.org/html/2608.18404#S5.SS1.p1.1),[§5\.3](https://arxiv.org/html/2608.18404#S5.SS3.p1.1)\.
- \[39\]R\. Young and N\. Pugeault\(2024\)Enhancing robustness in deep reinforcement learning: a lyapunov exponent approach\.Advances in Neural Information Processing Systems37,pp\. 86102–86123\.Cited by:[§2\.2](https://arxiv.org/html/2608.18404#S2.SS2.p1.1)\.
- \[40\]C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. Wu\(2022\)The surprising effectiveness of ppo in cooperative multi\-agent games\.Advances in neural information processing systems35,pp\. 24611–24624\.Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.2.1),[§1](https://arxiv.org/html/2608.18404#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.18404#S3.SS1.p1.3)\.
- \[41\]T\. Yu, Y\. Zhang, Z\. Zhang, and C\. M\. De Sa\(2022\)Understanding hyperdimensional computing for parallel single\-pass learning\.Advances in neural information processing systems35,pp\. 1157–1169\.Cited by:[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.18404#S3.SS2.p3.2)\.
- \[42\]S\. Yun, H\. Chen, R\. Masukawa, H\. Errahmouni Barkam, A\. Ding, W\. Huang, A\. Rezvani, S\. Angizi, and M\. Imani\(2024\)HyperSense: hyperdimensional intelligent sensing for energy\-efficient sparse data processing\.Advanced Intelligent Systems6\(12\),pp\. 2400228\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/aisy.202400228),[Link](https://advanced.onlinelibrary.wiley.com/doi/abs/10.1002/aisy.202400228),https://advanced\.onlinelibrary\.wiley\.com/doi/pdf/10\.1002/aisy\.202400228Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[43\]S\. Yun, H\. Oh, R\. Masukawa, P\. Mercati, N\. D\. Bastian, and M\. Imani\(2026\)LogHD: robust compression of hyperdimensional classifiers via logarithmic class\-axis reduction\.In2026 Design, Automation & Test in Europe Conference \(DATE\),pp\. 1–7\.Cited by:[§5\.3](https://arxiv.org/html/2608.18404#S5.SS3.p1.1)\.
- \[44\]S\. Zang, M\. Ding, D\. Smith, P\. Tyler, T\. Rakotoarivelo, and M\. A\. Kaafar\(2019\)The impact of adverse weather conditions on autonomous vehicles: how rain, snow, fog, and hail affect the performance of a self\-driving car\.IEEE Vehicular Technology Magazine14\(2\),pp\. 103–111\.External Links:[Document](https://dx.doi.org/10.1109/MVT.2019.2892497)Cited by:[§2\.2](https://arxiv.org/html/2608.18404#S2.SS2.p1.1)\.
- \[45\]Y\. Zhang, H\. Tang, H\. Lin, and W\. Ding\(2025\)Residual kernel policy network: enhancing stability and robustness in rkhs\-based reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Table 1](https://arxiv.org/html/2608.18404#S1.T1.fig1.2.1.4.1),[§2\.1](https://arxiv.org/html/2608.18404#S2.SS1.p1.1)\.
- \[46\]Q\. Zhao, A\. H\. Thomas, A\. Brin, X\. Yu, and T\. Rosing\(2025\)Bridging the gap between hyperdimensional computing and kernel methods via the nyström method\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 22813–22821\.Cited by:[§1](https://arxiv.org/html/2608.18404#S1.p2.1)\.
- \[47\]Z\. Zou, Y\. Kim, F\. Imani, H\. Alimohamadi, R\. Cammarota, and M\. Imani\(2021\)Scalable edge\-based hyperdimensional learning system with brain\-like neural adaptation\.InSC21: International Conference for High Performance Computing, Networking, Storage and Analysis,Vol\.,pp\. 1–15\.External Links:[Document](https://dx.doi.org/10.1145/3458817.3480958)Cited by:[§5\.3](https://arxiv.org/html/2608.18404#S5.SS3.p1.1)\.

## Appendix ABaseline Actor Architectures

DNN\.A two\-hidden\-layer MLP with ReLU activations maps observations to\|𝒜\|\|\\mathcal\{A\}\|action logits, followed by a softmax\. The hidden widths are tuned as described in[Appendix C](https://arxiv.org/html/2608.18404#A3)\. For classic control and MiniGrid, DNN uses the same REINFORCE advantages and Adam optimizer as VSPG\. On SustainGym, DNN \(Multi\-Agent\) uses decentralized actors and a shared centralized critic\. Each agent maintains its own actor, while a two\-hidden\-layer MLP critic \(default width 128\) maps the concatenated global state to a scalar value estimate\. Training uses GAE \(λ=0\.95\\lambda\{=\}0\.95\), a PPO\-style clipped objective \(ϵ=0\.2\\epsilon\{=\}0\.2\), and online normalization of critic targets\. The critic is trained on\-policy and discarded at evaluation\. Multi\-agent VSPG and Raw\-Linear use the same critic, advantages, and clipped importance\-ratio objective, isolating the actor representation rather than the training procedure\. DNN \(Single\) instead uses one actor and critic over the global state, with the actor producing a joint discretized action for all zones under the same training loop\.

Raw\-Linear\.A single linear layer maps the raw observation directly to\|𝒜\|\|\\mathcal\{A\}\|action logits, followed by a softmax, without hidden layers or HDC encoding\. Since VSPG is also linear in its encoded features,𝐂​ϕ​\(x\)\\mathbf\{C\}\\boldsymbol\{\\phi\}\(x\), this baseline isolates the effect of the hyperdimensional encoding from that of the linear policy\. All remaining training settings, including the SustainGym Multi\-Agent and Single variants, match DNN\.

QHD\.QHD\[[23](https://arxiv.org/html/2608.18404#bib.bib2)\]is the closest prior HDC baseline for single\-agent, discrete\-action RL\. It learns a linear hyperdimensionalQQ\-function,Q⁡\(s,a\)=M​\[a\]⊤​sQ\(s,a\)=M\[a\]^\{\\top\}s, using semi\-gradient Q\-learning:

Q⁡\(s,a\)←Q⁡\(s,a\)\+β⁡\[r\+γ​maxa′​Q−​\(s′,a′\)−Q⁡\(s,a\)\]Q\(s,a\)\\leftarrow Q\(s,a\)\+\\beta\\left\[r\+\\gamma\\max\_\{a^\{\\prime\}\}Q^\{\-\}\(s^\{\\prime\},a^\{\\prime\}\)\-Q\(s,a\)\\right\]\(19\)whereQ−Q^\{\-\}is a target copy hard\-synchronized at the interval reported astargetin[Table 3](https://arxiv.org/html/2608.18404#A3.T3)\. For the linear HDC representation,[Equation 19](https://arxiv.org/html/2608.18404#A1.E19)becomesM⁡\[a\]\+=β⁡\(qtrue−qpred\)​s,M\[a\]\\mathrel\{\+\}=\\beta\\left\(q\_\{\\mathrm\{true\}\}\-q\_\{\\mathrm\{pred\}\}\\right\)s,withqtrue=r\+γ​maxa′​Q−​\(s′,a′\)q\_\{\\mathrm\{true\}\}=r\+\\gamma\\max\_\{a^\{\\prime\}\}Q^\{\-\}\(s^\{\\prime\},a^\{\\prime\}\)andqpred=M​\[a\]⊤​sq\_\{\\mathrm\{pred\}\}=M\[a\]^\{\\top\}s\. Actions are selectedϵ\\epsilon\-greedily with linearly decayed exploration\. Because QHD is off\-policy whereas VSPG is on\-policy, we match episode budgets rather than update counts\. QHD is omitted from SustainGym because it is a single\-agent value\-based method and does not fit the actor\-focused multi\-agent comparison\.

## Appendix BEncoder Implementation Details Across Environments

The main paper defines a fixed base mapφe\\varphi\_\{e\}for each encoder familye∈\{Basis,FHRR,RFF\}e\\in\\\{\\mathrm\{Basis\},\\mathrm\{FHRR\},\\mathrm\{RFF\}\\\}\. Each environment\-specific encoder is obtained either by applyingφe\\varphi\_\{e\}directly to a preprocessed observation or by composing multipleφe\\varphi\_\{e\}\-encoded components through binding and bundling\.

Throughout this section,⊙\\odotdenotes binding and⊕\\oplusdenotes bundling\. These symbols refer to the corresponding VSA operations rather than to one shared scalar operation\. For the bipolar Basis encoder,⊙\\odotis element\-wise multiplication and⊕\\oplusis element\-wise addition\. For FHRR,⊙\\odotis element\-wise complex multiplication, equivalently phase addition, and⊕\\oplusis element\-wise complex addition\. Every resulting real\-valued hypervector is finally normalized to unit Euclidean norm before being passed to the actor\.

### B\.1MiniGrid: Compositional Encoding over Grid Cells

A MiniGrid observation consists of a7×7×37\\times 7\\times 3partial\-view image, containing an object, color, and state identifier for each cell, together with the agent direction\. Let𝒱⁡\(x\)\\mathcal\{V\}\(x\)denote the cells marked as visible in observationxx, and let\(ou​v,ku​v,qu​v\)\(o\_\{uv\},k\_\{uv\},q\_\{uv\}\)denote the object, color, and state identifiers at cell\(u,v\)\(u,v\)\. For the Basis and FHRR encoders, the raw observation representation has the common compositional form

ϕ~e​\(x\)=⨁\(u,v\)∈𝒱⁡\(x\)\[φepos​\(u,v\)⊙φeobj​\(ou​v\)⊙φecol​\(ku​v\)⊙φest​\(qu​v\)\]⊕φedir​\(d\),e∈\{Basis,FHRR\},\\widetilde\{\\boldsymbol\{\\phi\}\}\_\{e\}\(x\)=\\bigoplus\_\{\(u,v\)\\in\\mathcal\{V\}\(x\)\}\\left\[\\varphi\_\{e\}^\{\\mathrm\{pos\}\}\(u,v\)\\odot\\varphi\_\{e\}^\{\\mathrm\{obj\}\}\(o\_\{uv\}\)\\odot\\varphi\_\{e\}^\{\\mathrm\{col\}\}\(k\_\{uv\}\)\\odot\\varphi\_\{e\}^\{\\mathrm\{st\}\}\(q\_\{uv\}\)\\right\]\\oplus\\varphi\_\{e\}^\{\\mathrm\{dir\}\}\(d\),\\qquad e\\in\\\{\\mathrm\{Basis\},\\mathrm\{FHRR\}\\\},\(20\)whereddis the current direction\. Thus, each cell is represented by binding its position and categorical components, and the visible\-cell representations are bundled with the direction representation\.

##### Basis\.

MiniGrid uses the sign\-thresholded Basis map, so each categorical component is assigned a fixed bipolar hypervector\. Position is represented by binding independently drawn row and column encodings,

φBasispos​\(u,v\)=φBasisx​\(eu\)⊙φBasisy​\(ev\),\\varphi\_\{\\mathrm\{Basis\}\}^\{\\mathrm\{pos\}\}\(u,v\)=\\varphi\_\{\\mathrm\{Basis\}\}^\{x\}\(e\_\{u\}\)\\odot\\varphi\_\{\\mathrm\{Basis\}\}^\{y\}\(e\_\{v\}\),\(21\)whereeue\_\{u\}andeve\_\{v\}are one\-hot identifiers\. Object, color, state, and direction identifiers are encoded in the same way using independent fixed codebooks\. In[Equation 20](https://arxiv.org/html/2608.18404#A2.E20), binding is the Hadamard product and bundling is vector addition\. The resulting vector is then normalized to unit Euclidean norm\.

##### FHRR\.

For FHRR, the composition in[Equation 20](https://arxiv.org/html/2608.18404#A2.E20)is implemented using unit complex hypervectors\. LetDc=D/2D\_\{c\}=D/2and definecis⁡\(𝜽\)=cos⁡\(𝜽\)\+i​sin⁡\(𝜽\)\\operatorname\{cis\}\(\\boldsymbol\{\\theta\}\)=\\cos\(\\boldsymbol\{\\theta\}\)\+i\\sin\(\\boldsymbol\{\\theta\}\)\. The position encoder uses two Gaussian base\-phase vectors,

𝝍x,𝝍y∼𝒩⁡\(𝟎,w−2​𝐈Dc\),φFHRRpos​\(u,v\)=cis⁡\(u​𝝍x\+v​𝝍y\),\\boldsymbol\{\\psi\}\_\{x\},\\boldsymbol\{\\psi\}\_\{y\}\\sim\\mathcal\{N\}\\\!\\left\(\\mathbf\{0\},w^\{\-2\}\\mathbf\{I\}\_\{D\_\{c\}\}\\right\),\\qquad\\varphi\_\{\\mathrm\{FHRR\}\}^\{\\mathrm\{pos\}\}\(u,v\)=\\operatorname\{cis\}\\\!\\left\(u\\boldsymbol\{\\psi\}\_\{x\}\+v\\boldsymbol\{\\psi\}\_\{y\}\\right\),\(22\)wherewwcontrols the spatial kernel width\. Object, color, state, and direction identifiers use independent fixed phase codebooks,

𝝎r\(k\)∼𝒰​\(−π,π\)Dc,φFHRRr​\(k\)=cis⁡\(𝝎r\(k\)\),r∈\{obj,col,st,dir\}\.\\boldsymbol\{\\omega\}\_\{r\}^\{\(k\)\}\\sim\\mathcal\{U\}\(\-\\pi,\\pi\)^\{D\_\{c\}\},\\qquad\\varphi\_\{\\mathrm\{FHRR\}\}^\{r\}\(k\)=\\operatorname\{cis\}\\\!\\left\(\\boldsymbol\{\\omega\}\_\{r\}^\{\(k\)\}\\right\),\\quad r\\in\\\{\\mathrm\{obj\},\\mathrm\{col\},\\mathrm\{st\},\\mathrm\{dir\}\\\}\.\(23\)
The representation of visible cell\(u,v\)\(u,v\)is therefore

𝐳u​v\\displaystyle\\mathbf\{z\}\_\{uv\}=φFHRRpos​\(u,v\)⊙φFHRRobj​\(ou​v\)⊙φFHRRcol​\(ku​v\)⊙φFHRRst​\(qu​v\)\\displaystyle=\\varphi\_\{\\mathrm\{FHRR\}\}^\{\\mathrm\{pos\}\}\(u,v\)\\odot\\varphi\_\{\\mathrm\{FHRR\}\}^\{\\mathrm\{obj\}\}\(o\_\{uv\}\)\\odot\\varphi\_\{\\mathrm\{FHRR\}\}^\{\\mathrm\{col\}\}\(k\_\{uv\}\)\\odot\\varphi\_\{\\mathrm\{FHRR\}\}^\{\\mathrm\{st\}\}\(q\_\{uv\}\)\(24\)=cis⁡\(u​𝝍x\+v​𝝍y\+𝝎obj\(ou​v\)\+𝝎col\(ku​v\)\+𝝎st\(qu​v\)\)\.\\displaystyle=\\operatorname\{cis\}\\\!\\left\(u\\boldsymbol\{\\psi\}\_\{x\}\+v\\boldsymbol\{\\psi\}\_\{y\}\+\\boldsymbol\{\\omega\}\_\{\\mathrm\{obj\}\}^\{\(o\_\{uv\}\)\}\+\\boldsymbol\{\\omega\}\_\{\\mathrm\{col\}\}^\{\(k\_\{uv\}\)\}\+\\boldsymbol\{\\omega\}\_\{\\mathrm\{st\}\}^\{\(q\_\{uv\}\)\}\\right\)\.Thus, FHRR binding is implemented by element\-wise complex multiplication, which is equivalent to adding the component phases\.

The visible\-cell hypervectors are bundled by complex addition, after which the direction hypervector is added:

𝐳⁡\(x\)=⨁\(u,v\)∈𝒱⁡\(x\)𝐳u​v⊕φFHRRdir​\(d\)\.\\mathbf\{z\}\(x\)=\\bigoplus\_\{\(u,v\)\\in\\mathcal\{V\}\(x\)\}\\mathbf\{z\}\_\{uv\}\\oplus\\varphi\_\{\\mathrm\{FHRR\}\}^\{\\mathrm\{dir\}\}\(d\)\.\(25\)Following the implementation, each complex coordinate is then projected back to unit magnitude,

z^j\(x\)=zj​\(x\)max⁡\{\|zj​\(x\)\|,ε\},j=1,…,Dc,\\widehat\{z\}\_\{j\}\(x\)=\\frac\{z\_\{j\}\(x\)\}\{\\max\\\{\|z\_\{j\}\(x\)\|,\\varepsilon\\\}\},\\qquad j=1,\\ldots,D\_\{c\},\(26\)before the real and imaginary parts are interleaved intoϕ~FHRR​\(x\)∈ℝD\\widetilde\{\\boldsymbol\{\\phi\}\}\_\{\\mathrm\{FHRR\}\}\(x\)\\in\\mathbb\{R\}^\{D\}\. The interleaving is only a fixed coordinate permutation of the block real–imaginary representation and therefore preserves inner products\. The real\-valued output is finally normalized to unit Euclidean norm\.

The Gaussian position phases in[Equation 22](https://arxiv.org/html/2608.18404#A2.E22)induce a smooth spatial kernel\. For a displacement\(Δu,Δv\)\(\\Delta\_\{u\},\\Delta\_\{v\}\),

𝔼⁡\[1Dc​∑j=1Dccos⁡\(Δu​ψx,j\+Δv​ψy,j\)\]=exp⁡\(−Δu2\+Δv22​w2\)\.\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{D\_\{c\}\}\\sum\_\{j=1\}^\{D\_\{c\}\}\\cos\\\!\\left\(\\Delta\_\{u\}\\psi\_\{x,j\}\+\\Delta\_\{v\}\\psi\_\{y,j\}\\right\)\\right\]=\\exp\\\!\\left\(\-\\frac\{\\Delta\_\{u\}^\{2\}\+\\Delta\_\{v\}^\{2\}\}\{2w^\{2\}\}\\right\)\.\(27\)Hence,wwcontrols the spatial neighborhood over which observations share representation similarity\. Smaller values produce more nearly orthogonal positions, whereas larger values produce broader spatial generalization\. We usew=1\.0w=1\.0unless otherwise stated, giving expected similarityexp\(−1/2\)≈0\.61\\exp\(\-1/2\)\\approx 0\.61between horizontally or vertically adjacent cells\.

##### RFF\.

RFF is applied directly rather than compositionally:

ϕ~RFF​\(x\)=φRFF​\(TMG​\(x\)\),\\widetilde\{\\boldsymbol\{\\phi\}\}\_\{\\mathrm\{RFF\}\}\(x\)=\\varphi\_\{\\mathrm\{RFF\}\}\\\!\\left\(T\_\{\\mathrm\{MG\}\}\(x\)\\right\),\(28\)whereTMGT\_\{\\mathrm\{MG\}\}flattens the observation into a7×7×3\+1=1487\\times 7\\times 3\+1=148dimensional vector and rescales the object, color, state, and direction channels by1010,55,22, and33, respectively\. This prevents the raw categorical identifier ranges from determining the projection scale\. The output of[Equation 28](https://arxiv.org/html/2608.18404#A2.E28)is then normalized to unit Euclidean norm\.

### B\.2Classic Control: Direct Encoding

The classic\-control environments provide flat continuous observations:d=4d=4forCartPole\-v1,d=6d=6forAcrobot\-v1, andd=8d=8forLunarLander\-v2\. Since these observations contain no explicit compositional structure, each encoder is applied directly:

ϕ~e​\(x\)=φe​\(x\),e∈\{Basis,FHRR,RFF\}\.\\widetilde\{\\boldsymbol\{\\phi\}\}\_\{e\}\(x\)=\\varphi\_\{e\}\(x\),\\qquad e\\in\\\{\\mathrm\{Basis\},\\mathrm\{FHRR\},\\mathrm\{RFF\}\\\}\.\(29\)Basis uses the identity nonlinearity,φBasis​\(x\)=𝐖​x\\varphi\_\{\\mathrm\{Basis\}\}\(x\)=\\mathbf\{W\}x, while FHRR and RFF use their corresponding fixed random maps from the main paper\. The native observation ranges are already comparable, so no additional range normalization is applied before[Equation 29](https://arxiv.org/html/2608.18404#A2.E29)\. Every encoded observation is subsequently normalized to unit Euclidean norm\.

### B\.3SustainGym: Range\-Normalized Direct Encoding

SustainGym provides ad=10d=10flat observation whose components have substantially different physical scales\. We first apply component\-wise range normalization,

TSG​\(x\)=2​x−lohi−lo−1∈\[−1,1\]d,T\_\{\\mathrm\{SG\}\}\(x\)=2\\frac\{x\-\\mathrm\{lo\}\}\{\\mathrm\{hi\}\-\\mathrm\{lo\}\}\-1\\in\[\-1,1\]^\{d\},\(30\)and then apply the corresponding base map directly:

ϕ~e​\(x\)=φe​\(TSG​\(x\)\),e∈\{Basis,FHRR,RFF\}\.\\widetilde\{\\boldsymbol\{\\phi\}\}\_\{e\}\(x\)=\\varphi\_\{e\}\\\!\\left\(T\_\{\\mathrm\{SG\}\}\(x\)\\right\),\\qquad e\\in\\\{\\mathrm\{Basis\},\\mathrm\{FHRR\},\\mathrm\{RFF\}\\\}\.\(31\)The normalization in[Equation 30](https://arxiv.org/html/2608.18404#A2.E30)prevents large\-range quantities such as solar heat gain from dominating the random projection\. FHRR and RFF use their corresponding fixed maps, while Basis usesρ=sign\\rho=\\operatorname\{sign\}\. The output of[Equation 31](https://arxiv.org/html/2608.18404#A2.E31)is finally normalized to unit Euclidean norm\.

## Appendix CHyperparameter Tuning Budget

[Table 3](https://arxiv.org/html/2608.18404#A3.T3)reports the search space, number of evaluated configurations, and selected hyperparameters for each method and environment\. For VSPG,τ\\tauandη\\etaare tuned jointly becauseτ\\tauaffects both the logit scale and the effective update magnitude\. The baselines receive at least comparable tuning budgets: DNN and Raw\-Linear use similarly sized searches, while QHD is evaluated over a substantially larger grid\. Each selected configuration is then evaluated over five seeds for the results reported in the main paper\.

Table 3:Hyperparameter search spaces, tuning budgets, and selected configurations\. Each row reports one method–environment configuration \(and encoder for VSPG\)\. Repeated settings are shown so each row is self\-contained; fixed settings are not counted in thePts\.column\.MethodEnvironmentSearch/fixed settingsPts\.Chosen bestMiniGridVSPG \(Basis\)Empty\-5x5τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}16τ=10,η=5×10−3\\tau\{=\}10,\\eta\{=\}5\{\\times\}10^\{\-3\}VSPG \(Basis\)DoorKey\-5x5τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}16τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}VSPG \(Basis\)DoorKey\-8x8τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}16τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}VSPG \(FHRR\)Empty\-5x5τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},w=1\.0w\{=\}1\.016τ=10,η=10−2\\tau\{=\}10,\\eta\{=\}10^\{\-2\}VSPG \(FHRR\)DoorKey\-5x5τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},w=1\.0w\{=\}1\.016τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}VSPG \(FHRR\)DoorKey\-8x8τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},w=1\.0w\{=\}1\.016τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}VSPG \(RFF\)Empty\-5x5τ∈\{5,10,20\}\\tau\\in\\\{5,10,20\\\},η∈\{5×10−3,10−2,5×10−2\}\\eta\\in\\\{5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},σ∈\{0\.5,1\.0\}\\sigma\\in\\\{0\.5,1\.0\\\}18τ=20,η=10−2,σ=1\.0\\tau\{=\}20,\\eta\{=\}10^\{\-2\},\\sigma\{=\}1\.0VSPG \(RFF\)DoorKey\-5x5τ∈\{5,10,20\}\\tau\\in\\\{5,10,20\\\},η∈\{5×10−3,10−2,5×10−2\}\\eta\\in\\\{5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},σ∈\{0\.5,1\.0\}\\sigma\\in\\\{0\.5,1\.0\\\}18τ=10,η=5×10−3,σ=1\.0\\tau\{=\}10,\\eta\{=\}5\{\\times\}10^\{\-3\},\\sigma\{=\}1\.0VSPG \(RFF\)DoorKey\-8x8τ∈\{1,2,5,10\}\\tau\\in\\\{1,2,5,10\\\},η∈\{10−2,5×10−2\}\\eta\\in\\\{10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},σ∈\{0\.5,1\.0\}\\sigma\\in\\\{0\.5,1\.0\\\}16τ=5,η=5×10−2,σ=0\.5\\tau\{=\}5,\\eta\{=\}5\{\\times\}10^\{\-2\},\\sigma\{=\}0\.5DNNEmpty\-5x5lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=10−4\{=\}10^\{\-4\}, hidden=\[128,64\]\{=\}\[128,64\]DNNDoorKey\-5x5lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=3×10−4\{=\}3\{\\times\}10^\{\-4\}, hidden=\[256,128\]\{=\}\[256,128\]DNNDoorKey\-8x8lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=3×10−4\{=\}3\{\\times\}10^\{\-4\}, hidden=\[256,128\]\{=\}\[256,128\]Raw\-LinearEmpty\-5x5lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1\.0,2\.0,5\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0,5\.0\\\}16lr=10−3\{=\}10^\{\-3\},τ=0\.5\\tau\{=\}0\.5Raw\-LinearDoorKey\-5x5lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1\.0,2\.0,5\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0,5\.0\\\}16lr=10−3\{=\}10^\{\-3\},τ=5\.0\\tau\{=\}5\.0Raw\-LinearDoorKey\-8x8lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1\.0,2\.0,5\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0,5\.0\\\}16lr=10−3\{=\}10^\{\-3\},τ=5\.0\\tau\{=\}5\.0Classic controlVSPG \(FHRR\)CartPole\-v1τ∈\{10,20,40\}\\tau\\in\\\{10,20,40\\\},η∈\{10−6,10−5,10−4,10−3\}\\eta\\in\\\{10^\{\-6\},10^\{\-5\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}REINFORCE12τ=40,η=10−5\\tau\{=\}40,\\eta\{=\}10^\{\-5\}VSPG \(FHRR\)LunarLander\-v2τ∈\{10,20,40\}\\tau\\in\\\{10,20,40\\\},η∈\{10−6,10−5,10−4,10−3\}\\eta\\in\\\{10^\{\-6\},10^\{\-5\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}REINFORCE12τ=40,η=10−5\\tau\{=\}40,\\eta\{=\}10^\{\-5\}VSPG \(FHRR\)Acrobot\-v1τ∈\{5,7,10,20\}\\tau\\in\\\{5,7,10,20\\\},η∈\{5×10−4,10−4,10−3\}\\eta\\in\\\{5\{\\times\}10^\{\-4\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}GAE\+PPO\-clip12τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}VSPG \(Basis\)CartPole\-v1τ∈\{10,20,40\}\\tau\\in\\\{10,20,40\\\},η∈\{10−6,10−5,10−4,10−3\}\\eta\\in\\\{10^\{\-6\},10^\{\-5\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}REINFORCE12τ=40,η=10−5\\tau\{=\}40,\\eta\{=\}10^\{\-5\}VSPG \(Basis\)LunarLander\-v2τ∈\{10,20,40\}\\tau\\in\\\{10,20,40\\\},η∈\{10−6,10−5,10−4,10−3\}\\eta\\in\\\{10^\{\-6\},10^\{\-5\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}REINFORCE12τ=40,η=10−5\\tau\{=\}40,\\eta\{=\}10^\{\-5\}VSPG \(Basis\)Acrobot\-v1τ∈\{5,7,10,20\}\\tau\\in\\\{5,7,10,20\\\},η∈\{5×10−4,10−4,10−3\}\\eta\\in\\\{5\{\\times\}10^\{\-4\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}GAE\+PPO\-clip12τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}VSPG \(RFF\)CartPole\-v1τ∈\{10,20,40\}\\tau\\in\\\{10,20,40\\\},η∈\{10−6,10−5,10−4,10−3\}\\eta\\in\\\{10^\{\-6\},10^\{\-5\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}REINFORCE12τ=40,η=10−5\\tau\{=\}40,\\eta\{=\}10^\{\-5\}VSPG \(RFF\)LunarLander\-v2τ∈\{10,20,40\}\\tau\\in\\\{10,20,40\\\},η∈\{10−6,10−5,10−4,10−3\}\\eta\\in\\\{10^\{\-6\},10^\{\-5\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}REINFORCE12τ=40,η=10−5\\tau\{=\}40,\\eta\{=\}10^\{\-5\}VSPG \(RFF\)Acrobot\-v1τ∈\{5,7,10,20\}\\tau\\in\\\{5,7,10,20\\\},η∈\{5×10−4,10−4,10−3\}\\eta\\in\\\{5\{\\times\}10^\{\-4\},10^\{\-4\},10^\{\-3\}\\\}, advantage=\{=\}GAE\+PPO\-clip12τ=10,η=10−3\\tau\{=\}10,\\eta\{=\}10^\{\-3\}DNNCartPole\-v1lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=3×10−4\{=\}3\{\\times\}10^\{\-4\}, hidden=\[128,64\]\{=\}\[128,64\]DNNLunarLander\-v2lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=3×10−4\{=\}3\{\\times\}10^\{\-4\}, hidden=\[128,64\]\{=\}\[128,64\]DNNAcrobot\-v1lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=10−4\{=\}10^\{\-4\}, hidden=\[256,128\]\{=\}\[256,128\]Raw\-LinearCartPole\-v1lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1\.0,2\.0,5\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0,5\.0\\\}16lr=10−3\{=\}10^\{\-3\},τ=5\.0\\tau\{=\}5\.0Raw\-LinearLunarLander\-v2lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1\.0,2\.0,5\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0,5\.0\\\}16lr=10−3\{=\}10^\{\-3\},τ=5\.0\\tau\{=\}5\.0Raw\-LinearAcrobot\-v1lr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1\.0,2\.0,5\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0,5\.0\\\}16lr=10−3\{=\}10^\{\-3\},τ=2\.0\\tau\{=\}2\.0SustainGymVSPG \(Basis\)hot\_dryτ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}16τ=2,η=5×10−3\\tau\{=\}2,\\eta\{=\}5\{\\times\}10^\{\-3\}VSPG \(Basis\)warm\_humidτ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}16τ=1,η=5×10−3\\tau\{=\}1,\\eta\{=\}5\{\\times\}10^\{\-3\}VSPG \(FHRR\)hot\_dryτ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},w=1\.0w\{=\}1\.016τ=5,η=10−3\\tau\{=\}5,\\eta\{=\}10^\{\-3\}VSPG \(FHRR\)warm\_humidτ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},w=1\.0w\{=\}1\.016τ=2,η=5×10−3\\tau\{=\}2,\\eta\{=\}5\{\\times\}10^\{\-3\}VSPG \(RFF\)hot\_dryτ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},σ=1\.0\\sigma\{=\}1\.016τ=2,η=10−2\\tau\{=\}2,\\eta\{=\}10^\{\-2\}VSPG \(RFF\)warm\_humidτ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\},η∈\{10−3,5×10−3,10−2,5×10−2\}\\eta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\},σ=1\.0\\sigma\{=\}1\.016τ=1,η=5×10−3\\tau\{=\}1,\\eta\{=\}5\{\\times\}10^\{\-3\}DNNhot\_drylr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=10−4\{=\}10^\{\-4\}, hidden=\[128,64\]\{=\}\[128,64\]DNNwarm\_humidlr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\}, hidden∈\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\}\\in\\\{\[64,32\],\[128,64\],\[256,128\],\[256,256\]\\\}16lr=10−3\{=\}10^\{\-3\}, hidden=\[256,256\]\{=\}\[256,256\]Raw\-Linearhot\_drylr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\}16lr=3×10−4,τ=2\.0\{=\}3\{\\times\}10^\{\-4\},\\tau=2\.0Raw\-Linearwarm\_humidlr∈\{10−4,3×10−4,10−3,3×10−3\}\\in\\\{10^\{\-4\},3\{\\times\}10^\{\-4\},10^\{\-3\},3\{\\times\}10^\{\-3\}\\\},τ∈\{0\.5,1,2,5\}\\tau\\in\\\{0\.5,1,2,5\\\}16lr=3×10−3,τ=2\.0\{=\}3\{\\times\}10^\{\-3\},\\tau=2\.0Value\-basedQHDCartPole\-v1β∈\{10−3,5×10−3,10−2,5×10−2\}\\beta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}, batch∈\{2,4,10,32\}\\in\\\{2,4,10,32\\\}, target∈\{50,200\}\\in\\\{50,200\\\}, buffer∈\{2000,50000\}\\in\\\{2000,50000\\\}64β=5×10−2\\beta\{=\}5\{\\times\}10^\{\-2\}, batch=32\{=\}32, buffer=2000\{=\}2000, target=50\{=\}50QHDLunarLander\-v2β∈\{10−3,5×10−3,10−2,5×10−2\}\\beta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}, batch∈\{2,4,10,32\}\\in\\\{2,4,10,32\\\}, target∈\{50,200\}\\in\\\{50,200\\\}, buffer∈\{2000,50000\}\\in\\\{2000,50000\\\}64β=10−2\\beta\{=\}10^\{\-2\}, batch=32\{=\}32, buffer=50000\{=\}50000, target=50\{=\}50QHDAcrobot\-v1β∈\{10−3,5×10−3,10−2,5×10−2\}\\beta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}, batch∈\{2,4,10,32\}\\in\\\{2,4,10,32\\\}, target∈\{50,200\}\\in\\\{50,200\\\}, buffer∈\{2000,50000\}\\in\\\{2000,50000\\\}64β=10−3\\beta\{=\}10^\{\-3\}, batch=10\{=\}10, buffer=50000\{=\}50000, target=50\{=\}50QHDEmpty\-5x5β∈\{10−3,5×10−3,10−2,5×10−2\}\\beta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}, batch∈\{2,4,10,32\}\\in\\\{2,4,10,32\\\}, target∈\{50,200\}\\in\\\{50,200\\\}, buffer∈\{2000,50000\}\\in\\\{2000,50000\\\}64β=10−2\\beta\{=\}10^\{\-2\}, batch=10\{=\}10, buffer=2000\{=\}2000, target=200\{=\}200QHDDoorKey\-5x5β∈\{10−3,5×10−3,10−2,5×10−2\}\\beta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}, batch∈\{2,4,10,32\}\\in\\\{2,4,10,32\\\}, target∈\{50,200\}\\in\\\{50,200\\\}, buffer∈\{2000,50000\}\\in\\\{2000,50000\\\}64β=10−2\\beta\{=\}10^\{\-2\}, batch=32\{=\}32, buffer=50000\{=\}50000, target=200\{=\}200QHDDoorKey\-8x8β∈\{10−3,5×10−3,10−2,5×10−2\}\\beta\\in\\\{10^\{\-3\},5\{\\times\}10^\{\-3\},10^\{\-2\},5\{\\times\}10^\{\-2\}\\\}, batch∈\{2,4,10,32\}\\in\\\{2,4,10,32\\\}, target∈\{50,200\}\\in\\\{50,200\\\}, buffer∈\{2000,50000\}\\in\\\{2000,50000\\\}64β=10−2\\beta\{=\}10^\{\-2\}, batch=32\{=\}32, buffer=50000\{=\}50000, target=200\{=\}200
## Appendix DClosed\-Form Equivalence and the Sphere Constraint

![Refer to caption](https://arxiv.org/html/2608.18404v1/paper_row_summary_last.png)Figure 6:Closed\-form and directly differentiated VSPG updates under standard unit\-norm initialization \(top\) and zero initialization \(bottom\)\. Colors denote the encoder; solid, dashed, and dotted lines indicate closed\-form, normalized SGD, and unnormalized SGD, respectively\. Shading shows±\\pmSEM over five seeds\. Standard initialization yields broadly similar behavior, whereas zero initialization reveals sensitivity to sphere projection\.VSPG can be viewed both as a vector\-symbolic actor trained by advantage\-weighted bundling and as a log\-linear softmax policy over fixed HDC features\. Proposition 1 connects these views by showing that the bundling termΛ⊤​S\\Lambda^\{\\top\}Sis exactly the sampled policy gradient of the softmax surrogate\. We examine whether the closed\-form implementation reproduces direct differentiation of the same objective, and how its random unit\-norm initialization and row\-wise sphere projection affect learning\.

We compare the closed\-form VSPG update with directly differentiated variants with and without row normalization\. Each is evaluated under standard Gaussian unit\-norm initialization and exact zero initialization of the action\-hypervector matrix𝐂\\mathbf\{C\}\. Zero initialization removes the random initialization term in Proposition 2, leaving action vectors formed entirely from accumulated policy\-gradient evidence\. All configurations are independently tuned and evaluated over five seeds across six environments and three encoders\.

Under standard initialization, the differentiated variants generally exhibit learning behavior similar to the closed\-form implementation \(Figure[6](https://arxiv.org/html/2608.18404#A4.F6), top\), consistent with Proposition 1\. Larger differences appear on the harder DoorKey tasks, reflecting projection, parameter\-norm dynamics, and independently tuned optimization scales rather than a different policy objective\.

Zero initialization reveals a stronger interaction with row normalization \(Figure[6](https://arxiv.org/html/2608.18404#A4.F6), bottom\)\. At𝐂=0\\mathbf\{C\}=0, the policy is initially uniform, and the first update contains only evidence from the initial trajectories\. Immediate normalization maps this update to unit norm regardless of its magnitude, allowing weak or noisy early evidence to determine a full\-scale action direction\. Without normalization, the action\-vector norms instead grow gradually with the accumulated gradient signal, and these variants continue to learn in several settings where the normalized versions remain weak\.

Thus, neither nonzero initialization nor row normalization is required for the policy\-gradient identity itself\. Row normalization enforces the bounded cosine\-policy parameterization and fixed\-scale action\-memory geometry analyzed in the main paper, but can amplify early updates when combined with zero initialization\. Overall, the differentiated results support the closed\-form implementation as an algebraic realization of policy\-gradient learning over fixed HDC features, while exposing a practical interaction between initialization and the sphere constraint\.