Reinforcement Learning with Complex (valued) Memories

# Reinforcement Learning with Complex (valued) Memories
Source: [https://arxiv.org/html/2609.38598](https://arxiv.org/html/2609.38598)
Sathya Kamesh Bhethanabhotla††thanks:Work done partially at the University of Freiburg\.Efstratios GavvesAffiliation:University of AmsterdamAndré Biedenkapp11footnotemark:1Affiliation:Karlsruhe Institute of TechnologyEmail:[biedenkapp@kit\.edu](mailto:)

###### Abstract

Partially observable environments pose a fundamental challenge in deep reinforcement learning, requiring agents to compress temporal information from observations and maintain a memory to make effective decisions\. While there exist many approaches ranging from gated recurrence to attention mechanisms and model\-based RL, the search for effective representational techniques that can capture long\-term dependencies remains an active area of research\. In this work we revisit Unitary recurrent networks \(uRNNs\)\([Arjovsky et al\., 2016](https://arxiv.org/html/2609.38598#bib.bib7);[Jing et al\., 2017c](https://arxiv.org/html/2609.38598#bib.bib11)\), that demonstrated superior gradient flow and associative recall, expressing the recurrence and the hidden state in a complex vector space\. Their norm preserving unitary dynamics enable information propagation through long sequences\. To this end, we propose three different versions of uRNNs as drop\-in replacements for recurrent PPO architectures, and demonstrate that the simple recurrence and the added degree of freedom from the phase of the complex representations enable significant gains over baselines on several memory\-improvable tasks, including continuous control\. We further explore how to preserve the phase information of the complex hidden state for aphase\-awarepolicy by drawing a parallel to how quantum states are measured\. With our methods reaching up to22\-3×3\\timesthe reward in environments like rocksample and Craftax compared to the baselines, this work points towards an exciting new direction of representations for RL and the problem of partial observability\. Code is available at:[https://github\.com/Sathya98/qurl](https://github.com/Sathya98/qurl)

## 1Introduction

The human mind fundamentally relies on memory to function effectively in our partially observable world\. When navigating through an environment, we do not have access to complete information about our surroundings at any given moment\. Instead, we must integrate observations over time, maintaining an internal representation of relevant information from the past to inform our decisions and actions\. Memory is what makes action possible\. Without it, we could neither track a ball in motion, recall where we left our keys hours earlier nor any other task that unfolds over time\. In our effort to understand and replicate the mechanics of sequential decision\-making, we have turned to frameworks like reinforcement learning\([Sutton and Barto, 1998](https://arxiv.org/html/2609.38598#bib.bib50), RL; \), developing methods that can emulate these aspects of agency\.

Deep RL has addressed partial observability\([Åström, Karl Johan, 1965](https://arxiv.org/html/2609.38598#bib.bib10);[Kaelbling et al\., 1998](https://arxiv.org/html/2609.38598#bib.bib55)\)and memory in a variety of ways in the past decades of work\([Lambrechts et al\., 2022](https://arxiv.org/html/2609.38598#bib.bib56);[Ni et al\., 2022](https://arxiv.org/html/2609.38598#bib.bib57), see, e\.g\.\)\. We have also seen the development of various environments that can be used to test different aspects of these problems\. From early work on value\-based deep learning methods like DQN\([Mnih et al\., 2015](https://arxiv.org/html/2609.38598#bib.bib14)\)and the recurrent DRQN\([Hausknecht and Stone, 2015](https://arxiv.org/html/2609.38598#bib.bib15)\)playing Atari games\([Bellemare et al\., 2013](https://arxiv.org/html/2609.38598#bib.bib34)\), over policy gradient methods like Trust Region Policy Optimization \(TRPO\)\([Schulman et al\., 2017a](https://arxiv.org/html/2609.38598#bib.bib8)\)and Proximal Policy Optimization \(PPO\)\([Schulman et al\., 2017b](https://arxiv.org/html/2609.38598#bib.bib1)\), to transformer\-based methods\([Chen et al\., 2021](https://arxiv.org/html/2609.38598#bib.bib17)\)we have seen a lot of progress in the field\. Further, model\-based approaches such as Dreamer\([Hafner et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib36)\), PETS\([Chua et al\., 2018](https://arxiv.org/html/2609.38598#bib.bib58)\)or Recall2Imagine\([Samsami et al\., 2024](https://arxiv.org/html/2609.38598#bib.bib6)\)build an internal models of the world and use those to plan, enabling them to tackle partial observability\. With techniques like gated recurrence\([Hochreiter and Schmidhuber, 1997](https://arxiv.org/html/2609.38598#bib.bib19)\), attention\([Vaswani et al\., 2017](https://arxiv.org/html/2609.38598#bib.bib31)\), and state space models\([Gu and Dao, 2024](https://arxiv.org/html/2609.38598#bib.bib41)\), the search for effective representational techniques to compress state information over time remains an active area of research\. Despite this progress, the field still struggles with learning instabilities, for example caused by vanishing gradients or the dependence on short horizons\.

A decade ago, Unitary Evolution Recurrent Neural Networks\([Arjovsky et al\., 2016](https://arxiv.org/html/2609.38598#bib.bib7), uRNNs;\)offered a fresh approach to addressing the challenges of memory and gradient flow in recurrent architectures\. uRNNs operate in a complex\-valued vector space, using complex\-valued parameters and hidden states\. The central insight of uRNNs is that constraining the recurrent dynamics to be*unitary*, a norm\-preserving transformation that ensures stable evolution of the hidden state over arbitrarily long sequences, and aids associative recall\. This extension to the complex domain provides an additional degree of freedom in the form of a phase, allowing for richer representational capacity\. In this work, we propose this class of recurrence as a representation paradigm for RL agents acting in partially observable tasks, requiring long term memory\. We show how different variants of this recurrent class can be straightforwardly integrated into the recurrent PPO method\. Instead of learning real\-valued representations these drop\-in replacements enable learning complex\-valued representations\. We empirically show that such complex\-valued representations already achieve a significantly better performance across a range of memory improvable tasks\.

While complex\-valued representations already enable PPO to learn better policies, we further aim to leverage the phase information directly\. We propose a novel approach to propagate the phases captured in the complex hidden state to a discrete stochastic policy\. By allowing the phases of the complex parameters to interfere constructively and destructively through a Born\-rule like sampling\([Born, 1984](https://arxiv.org/html/2609.38598#bib.bib59)\), akin to collapsing a wavefunction, we highlight a potentially deeper connection between policies and quantum states\. Evaluating thisPhase Awarepolicy shows competitive performance to other baselines over tasks like T\-maze, battleship and craftax\. Particularly in long\-horizon tasks, our approach outperforms canonical recurrent architectures\. Our contributions are:

1. I\)We propose a recurrent PPO approach that uses the uRNN as the recurrent cell;
2. II\)We extend the uRNN architecture to enable an input\-dependent unitary transformation matrix;
3. III\)We propose a way to interpret the policy from a quantum information perspective, and introduce a completely complex\-valued policy head with a sampling method inspired by the Born rule in quantum mechanics;
4. IV\)We evaluate the proposed methods on a range of partially observable tasks, and show that the uRNN architecture is capable of solving these tasks and outperforms the baselines\.

## 2Background

### 2\.1Partial Observability, Memory and the problem setting

In many real\-world applications, the state is only partially observable, formalized as a Partially Observable Markov Decision Process \(POMDP\)\([Åström, Karl Johan, 1965](https://arxiv.org/html/2609.38598#bib.bib10)\)\. A POMDP augments the standard Markov Decision Process with an observation model and is described by the tuple\(𝒮,𝒜,T,R,Ω,O,γ\)\(\\mathcal\{S\},\\mathcal\{A\},T,R,\\Omega,O,\\gamma\), where𝒮\\mathcal\{S\}is the set of states and𝒜\\mathcal\{A\}the set of actions;T⁡\(st\+1∣st,at\)T\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\)is the state transition distribution;R⁡\(st,at\)R\(s\_\{t\},a\_\{t\}\)is the reward function;Ω\\Omegais the set of observations andO⁡\(ot∣st\)O\(o\_\{t\}\\mid s\_\{t\}\)the observation model;γ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\.

Unlike the fully observable case, the agent never accesses the underlying statests\_\{t\}directly; at each step it receives only an observationot∈Ωo\_\{t\}\\in\\Omegaemitted fromsts\_\{t\}throughOO\. A single observation is in general non\-Markovian, in that it does not on its own summarize the information needed to act optimally, so a policyπ⁡\(at∣ot\)\\pi\(a\_\{t\}\\mid o\_\{t\}\)conditioned on the latest observation alone is insufficient\. To recover the optimal policy, the agent must instead estimate the latent state from the history of observations seen so far,o1:t=\(o1,o2,…,ot\)o\_\{1:t\}=\(o\_\{1\},o\_\{2\},\\dots,o\_\{t\}\), compressing this history into a representation sufficient for action selection\. This introduces the need for agents to maintain memory of past observations, and it is the partially observable setting we operate in throughout this work\.

The simplest approach is frame stacking\([Bellemare et al\., 2013](https://arxiv.org/html/2609.38598#bib.bib34)\), also employed by[Mnih et al\. \(2015\)](https://arxiv.org/html/2609.38598#bib.bib14)in DQN for Atari, where the last four frames were stacked as input\. However, this captures dependencies only within the fixed window\. Recurrent networks address this by compressing temporal information into a learned hidden state\. DRQN\([Hausknecht and Stone, 2015](https://arxiv.org/html/2609.38598#bib.bib15)\)replaced frame stacking with LSTM layers that process single frames while maintaining a hidden state, matching DQN’s performance while being more robust to partial observability\.

Recurrent architectures still face limitations: sequential processing prevents parallelization and fixed\-size hidden states bottleneck information flow\. GTrXL\([Parisotto et al\., 2020](https://arxiv.org/html/2609.38598#bib.bib16)\)stabilized Transformers for RL through gating mechanisms\. Decision Transformer\([Chen et al\., 2021](https://arxiv.org/html/2609.38598#bib.bib17)\)reframed RL as sequence modeling by conditioning on desired returns\.

### 2\.2Unitary Evolution Recurrent Neural Networks

Recurrent neural networks suffer from well\-known stability issues: when the recurrent weight matrix has eigenvalues with magnitude not equal to one, the hidden state dynamics either contract or blow up exponentially, leading to vanishing and exploding gradients\([Bengio et al\., 1994](https://arxiv.org/html/2609.38598#bib.bib18)\)\. Gated architectures such as LSTMs\([Hochreiter and Schmidhuber, 1997](https://arxiv.org/html/2609.38598#bib.bib19)\)mitigate this through learned gating mechanisms, but still lack an inductive bias to bound the gradients\.

The introduction of*Unitary Evolution Recurrent Neural Networks*\(uRNNs\) by[Arjovsky et al\. \(2016\)](https://arxiv.org/html/2609.38598#bib.bib7)directly targeted this gap\. The central insight is that if the recurrent transition is constrained to be*unitary*, then the hidden state evolves through a norm\-preserving transformation at every time step\. A unitary matrixU∈ℂn×nU\\in\\mathbb\{C\}^\{n\\times n\}, satisfyingU​U†=U†​U=IUU^\{\\dagger\}=U^\{\\dagger\}U=I, is the complex analog of an orthogonal matrixQ∈ℝn×nQ\\in\\mathbb\{R\}^\{n\\times n\}\(for whichQ​Q⊤=IQQ^\{\\top\}=I\)\. Refer to[B](https://arxiv.org/html/2609.38598#A2)for an involved introduction\.

##### Unitary Recurrent Dynamics\.

A uRNN maintains a hidden stateht∈ℂnh\_\{t\}\\in\\mathbb\{C\}^\{n\}that evolves as:

ht\+1=σ⁡\(U​ht\+Vx​xt\),h\_\{t\+1\}=\\sigma\(Uh\_\{t\}\+V\_\{x\}x\_\{t\}\),\(1\)whereU∈ℂn×nU\\in\\mathbb\{C\}^\{n\\times n\}is a unitary matrix,VxV\_\{x\}maps real\-valued inputs into the complex hidden space, andσ\\sigmais the modReLU activation:

σmodReLU​\(z\)=\{\(\|z\|\+b\)​z\|z\|if​\|z\|\+b≥0,0if​\|z\|\+b<0\.\\sigma\_\{\\text\{modReLU\}\}\(z\)=\\begin\{cases\}\(\|z\|\+b\)\\frac\{z\}\{\|z\|\}&\\text\{if \}\|z\|\+b\\geq 0,\\\\ 0&\\text\{if \}\|z\|\+b<0\.\\end\{cases\}\(2\)

##### Efficient Parameterization\.

The unitary matrix is decomposed into structured factors:

U=D3​R2​ℱ−1​D2​Π​R1​ℱ​D1,U=D\_\{3\}\\,R\_\{2\}\\,\\mathcal\{F\}^\{\-1\}\\,D\_\{2\}\\,\\Pi\\,R\_\{1\}\\,\\mathcal\{F\}\\,D\_\{1\},\(3\)whereD1,D2,D3D\_\{1\},D\_\{2\},D\_\{3\}are diagonal phase matrices,ℱ\\mathcal\{F\}is the DFT,R1,R2R\_\{1\},R\_\{2\}are Householder reflections, andΠ\\Piis a fixed random permutation\. Each component is unitary by construction, givingO⁡\(n​log⁡n\)O\(n\\log n\)complexity\. Gradient updates are performed on unrestricted parameters, guaranteeing unitarity without projection\. uRNNs demonstrated remarkable performance on associative recall tasks, outperforming LSTMs with a fraction of the parameters\. This makes the architecture appealing for RL, where memory\-related tasks require integrating information over long sequences\.

## 3Related Work

#### Benchmarking Memory and Reinforcement Learning

Works such as[Hausknecht and Stone \(2015\)](https://arxiv.org/html/2609.38598#bib.bib15);[Parisotto et al\. \(2020\)](https://arxiv.org/html/2609.38598#bib.bib16);[Chen et al\. \(2021\)](https://arxiv.org/html/2609.38598#bib.bib17)presented methods relying on recurrence and transformers to handle partial observability\. Notably,[Ni et al\. \(2023\)](https://arxiv.org/html/2609.38598#bib.bib32)showed that transformers can be effective in capturing memory but are unable to assign credit correctly to past actions\. Memory Gym\([Pleines et al\., 2023](https://arxiv.org/html/2609.38598#bib.bib13)\)and POPGym\([Morad et al\., 2023](https://arxiv.org/html/2609.38598#bib.bib33)\)provide benchmark suites of partially observable environments requiring memory\. RLBenchNet\([Smirnov and Gu, 2025](https://arxiv.org/html/2609.38598#bib.bib12)\)presents an evaluation suite of PPO agents across various architectures\.

In our work, we use the POBAX benchmark\([Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\)for our experiments, baselines and comparison\. POBAX a benchmark for a suite of partially observable and memory improvable tasks written in JAX\([Bradbury et al\., 2018](https://arxiv.org/html/2609.38598#bib.bib5)\), including tasks ranging from vector input continuous control to complicated image based tasks like Crafter\([Hafner, 2021](https://arxiv.org/html/2609.38598#bib.bib25)\)\. They compare the performance of a GRU based recurrent PPO and a Lambda Discrepancy PPO\([Allen et al\., 2024](https://arxiv.org/html/2609.38598#bib.bib54)\)method against a memoryless and a full state baseline\.

#### Model\-Based Reinforcement Learning

Model\-Based RL relies on learning a world model\([Ha and Schmidhuber, 2018](https://arxiv.org/html/2609.38598#bib.bib37)\)that can represent the state space of partially observable environments\. These models can learn the dynamics of the environment in terms of this compressed latent statep⁡\(st\+1\|st,at\)p\(s\_\{t\+1\}\|s\_\{t\},a\_\{t\}\), enabling planning and off policy RL methods like SAC\([Haarnoja et al\., 2017](https://arxiv.org/html/2609.38598#bib.bib52)\), and TD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2609.38598#bib.bib51)\)\. Most notable are PlaNet and Dreamer\([Hafner et al\., 2019](https://arxiv.org/html/2609.38598#bib.bib35);[Hafner et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib36)\), employing RSSM\-based GRU cells\. Recall2Imagine\([Samsami et al\., 2024](https://arxiv.org/html/2609.38598#bib.bib6)\)proposed a world model based on S4\([Gu et al\., 2021](https://arxiv.org/html/2609.38598#bib.bib38)\)for long\-term memory\. Interestingly, S4 and Mamba\([Gu and Dao, 2024](https://arxiv.org/html/2609.38598#bib.bib41)\)are based on the same kind of recurrence as the uRNN, differing only in the applied non\-linearity\.

#### Complex\-Valued Neural Networks

Neural networks with complex parameters have been explored for many years\.[Amin et al\. \(2011\)](https://arxiv.org/html/2609.38598#bib.bib46)showed how Wirtinger calculus enables gradient descent for complex\-valued networks\. Follow\-up works\([Trabelsi et al\., 2018](https://arxiv.org/html/2609.38598#bib.bib29);[Sarroff et al\., 2015](https://arxiv.org/html/2609.38598#bib.bib47);[Geuchen and Voigtlaender, 2023](https://arxiv.org/html/2609.38598#bib.bib48)\)demonstrated capabilities across domains\. For a comprehensive overview, see[Bassey et al\. \(2021\)](https://arxiv.org/html/2609.38598#bib.bib30)\.

#### Successors of uRNNs

Several works addressed the limitation that the original uRNN’s parameterization does not span the entire unitary group\.[Wisdom et al\. \(2016\)](https://arxiv.org/html/2609.38598#bib.bib20)proposed full\-capacity unitary RNNs on Stiefel manifolds\.[Mhammedi et al\. \(2017\)](https://arxiv.org/html/2609.38598#bib.bib26)parameterized real orthogonal matrices via Householder reflections \(oRNN\)\.[Helfrich et al\. \(2018\)](https://arxiv.org/html/2609.38598#bib.bib27)introduced scoRNN via the scaled Cayley transform\.[Jing et al\. \(2017a\)](https://arxiv.org/html/2609.38598#bib.bib28)extended unitary RNNs with gating in GORU, addressing the lack of an explicit forget mechanism\. Later, the LRU\([Orvieto et al\., 2023](https://arxiv.org/html/2609.38598#bib.bib60)\)was introduced as a linear RNN with complex hidden states, which was the extended in[Elelimy et al\. \(2024\)](https://arxiv.org/html/2609.38598#bib.bib61)for real\-time RL\. We use the original uRNN architecture as a starting point for exploring unitary transformations in RL\.

In this work, we particularly concentrate on the method presented in\([Jing et al\., 2017c](https://arxiv.org/html/2609.38598#bib.bib11)\), a tuneable version of the uRNN composed of unitary transforms on alternate dimensions, that can span the whole unitary space for a given dimensionality\. We explore this construction as an additional method for complex\-valued recurrence, owing to its simple design and tuneable expressivity\.

## 4Unitary Recurrent Networks for PPO

We equip a PPO agent with a unitary recurrent cell, hypothesizing that a complex hidden state evolved by norm\-preserving transformations provides a stable and expressive memory for partially observable control\. Complex data types, forward and backward passes are fully supported in JAX and PyTorch, which we rely on, for building the method \(see Appendix[A](https://arxiv.org/html/2609.38598#A1)\)\.

### 4\.1Integrating the uRNN into Proximal Policy Optimization

We build on the recurrent PPO implementation of POBax\([Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\), where the base recurrent cell is a GRU\([Cho et al\., 2014](https://arxiv.org/html/2609.38598#bib.bib4)\)\. A task\-specific encoder maps each observationoto\_\{t\}to a feature vectorxtx\_\{t\}, which drives a recurrent cell whose hidden statehth\_\{t\}feeds separate policy and value heads\.

observationoto\_\{t\}Encoderxtx\_\{t\}GRU Cellrealht∈ℝnh\_\{t\}\\in\\mathbb\{R\}^\{n\}Unitary Recurrent Cellcomplexht∈ℂnh\_\{t\}\\in\\mathbb\{C\}^\{n\}ht−1h\_\{t\-1\}ht−1h\_\{t\-1\}Policy HeadValue HeadPolicy HeadValue Headhth\_\{t\}hth\_\{t\}π⁡\(at∣ht\)\\pi\(a\_\{t\}\\mid h\_\{t\}\)π⁡\(at∣ht\)\\pi\(a\_\{t\}\\mid h\_\{t\}\)v⁡\(ht\)v\(h\_\{t\}\)v⁡\(ht\)v\(h\_\{t\}\)GRU\-PPOuRNN\-PPO

Figure 1:Unitary Recurrent PPO architectures\.The Unitary Recurrent cell can be simply dropped\-in for any standard recurrent cell\. We show how the different variants can be constructed with complex hidden states, and later, fully complex policies\.We replace this GRU with a unitary recurrent cell carrying a complex hidden stateht∈ℂnh\_\{t\}\\in\\mathbb\{C\}^\{n\}, wherennmatches the GRU baseline’s per\-environment hidden size \(Table[2](https://arxiv.org/html/2609.38598#A5.T2)\), leaving the rest of the agent unchanged \(Figure[1](https://arxiv.org/html/2609.38598#S4.F1)\)\. We consider two parameterizations of the unitary transitionUU\. In its original form\([Arjovsky et al\., 2016](https://arxiv.org/html/2609.38598#bib.bib7)\), the recurrence follows Equation \([1](https://arxiv.org/html/2609.38598#S2.E1)\) withUUstructured as in Equation \([3](https://arxiv.org/html/2609.38598#S2.E3)\):UUis constant across time and the input enters only throughVxV\_\{x\}\. Nativecomplex64support allows a much simpler implementation than the original; the components are:

- •D1,D2,D3D\_\{1\},D\_\{2\},D\_\{3\}: Diagonal matrices parameterized with learnable phases:Dj,j=ei​wj,wj∈ℝD\_\{j,j\}=e^\{iw\_\{j\}\},\\;w\_\{j\}\\in\\mathbb\{R\}\.
- •R1,R2R\_\{1\},R\_\{2\}: Reflection matrices with learnablev1,v2∈ℂnv\_\{1\},v\_\{2\}\\in\\mathbb\{C\}^\{n\}:Ri=I−2​vk​vk∗/‖vk‖2R\_\{i\}=I\-2v\_\{k\}v\_\{k\}^\{\\ast\}/\\\|v\_\{k\}\\\|^\{2\}\.
- •VxV\_\{x\}: Acomplex64linear layer mapping the feature vector toℂn\\mathbb\{C\}^\{n\}\(no bias\)\.
- •modReLU: Implemented as per Equation \([2](https://arxiv.org/html/2609.38598#S2.E2)\)\.

The policy and value networks receive2​n2ninput features \(real and imaginary parts of the hidden state\)\. This construction, due to its time/step independent recurrent dynamics, takes up far fewer parameters than a GRU cell, even though the complex parameters use twice the floating point space\.

#### Parameter Initialization\.

We follow the original initialization\([Arjovsky et al\., 2016](https://arxiv.org/html/2609.38598#bib.bib7)\):DDphases from𝒰⁡\(−π,π\)\\mathcal\{U\}\(\-\\pi,\\pi\), reflection vectors from𝒰⁡\(−1,1\)\\mathcal\{U\}\(\-1,1\)for both components,VxV\_\{x\}via Glorot initialization\([Glorot and Bengio, 2010](https://arxiv.org/html/2609.38598#bib.bib49)\), and modReLU bias initialized to 0\.

#### Hidden State Initialization\.

We propose initializing the hidden state with a constant complex vector such that‖h0‖2=1\\\|h\_\{0\}\\\|\_\{2\}=1:

h0​\[k\]=12​n​\(1\+i\)k∈\{1,2,…,n\}h\_\{0\}\[k\]=\\frac\{1\}\{\\sqrt\{2n\}\}\\left\(1\+i\\right\)\\quad k\\in\\\{1,2,\\ldots,n\\\}\(4\)This is equivalent to an equal superposition state in annn\-dimensional eigenbasis \(Appendix[D](https://arxiv.org/html/2609.38598#A4)\), pointing to an interesting interpretation explored in Section[6](https://arxiv.org/html/2609.38598#S6)\.

### 4\.2Tunable Unitary Recurrence with EUNNs

The original parameterization \(Equation \([3](https://arxiv.org/html/2609.38598#S2.E3)\)\) is efficient but spans only a subset of the unitary group\([Wisdom et al\., 2016](https://arxiv.org/html/2609.38598#bib.bib20)\)\. As a second method we adopt the Efficient Unitary Neural Network \(EUNN\) of[Jing et al\. \(2017c\)](https://arxiv.org/html/2609.38598#bib.bib11), replacing the fixed FFT structure with a product ofLLlayers whose expressivity is directly tunable:

UEUNN=DFLFL−1⋯F1,U\_\{\\text\{EUNN\}\}=D\\,F\_\{L\}F\_\{L\-1\}\\cdots F\_\{1\},\(5\)whereDDis a diagonal phase and eachFkF\_\{k\}is block\-diagonal inH/2H/2independent2×22\\times 2unitary rotations, each acting on a pair of hidden dimensions\(a,b\)\(a,b\):

\(ha′hb′\)=\(ei​ϕ​cos⁡θ−ei​ϕ​sin⁡θsin⁡θcos⁡θ\)​\(hahb\),\\begin\{pmatrix\}h^\{\\prime\}\_\{a\}\\\\ h^\{\\prime\}\_\{b\}\\end\{pmatrix\}=\\begin\{pmatrix\}e^\{i\\phi\}\\cos\\theta&\-e^\{i\\phi\}\\sin\\theta\\\\\[2\.0pt\] \\sin\\theta&\\cos\\theta\\end\{pmatrix\}\\begin\{pmatrix\}h\_\{a\}\\\\ h\_\{b\}\\end\{pmatrix\},\(6\)with per\-pair anglesθ,ϕ\\theta,\\phi\. Consecutive layers alternate which dimensions are paired via a cyclic shift, mixing information across the full state\. The capacityLLtunes expressivity: smallLLrestrictsUUto a structured subspace, whileL=HL=Hprovably spans the full unitary groupU⁡\(H\)U\(H\)\. The recurrence is again Equation \([1](https://arxiv.org/html/2609.38598#S2.E1)\) withU=UEUNNU=U\_\{\\text\{EUNN\}\}, using learnable, input\-independent angles\.

### 4\.3Conditioning the Unitary Transformation on the Observation

While a constantUUensures stable gradient flow and is parameter\-efficient, we hypothesize that an observation\-dependentU⁡\(xt\)U\(x\_\{t\}\)can improve representational capacity\. This mirrors the selectivity of modern state\-space models, where the transition is made input\-dependent\([Katharopoulos et al\., 2020](https://arxiv.org/html/2609.38598#bib.bib42);[Gu and Dao, 2024](https://arxiv.org/html/2609.38598#bib.bib41);[Yang et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib43);[Yang et al\., 2024](https://arxiv.org/html/2609.38598#bib.bib44);[Siems et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib45)\)\.

We propose the ICuRNN \(Input Conditional Unitary RNN\) that conditionsUUon the inputxtx\_\{t\}:

- •D1,D2,D3D\_\{1\},D\_\{2\},D\_\{3\}: Now parameterized with a real\-valued linear layer:Dj=Linear​\(xt\)D\_\{j\}=\\text\{Linear\}\(x\_\{t\}\)\.
- •R1,R2R\_\{1\},R\_\{2\}: Parameterized with acomplex64linear layer:vk=ComplexLinear​\(xt\)v\_\{k\}=\\text\{ComplexLinear\}\(x\_\{t\}\)\.

The remaining components \(VxV\_\{x\}, hidden state initialization\) are unchanged\. While this adds more parameters, we hypothesize that it allows the policy to learn what to remember faster\.

## 5Do uRNNs improve memory capabilities?

### 5\.1Environments

We evaluate on seven tasks from the POBax suite\([Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\), each partially observable but solvable with the right memory, spanning discrete recall, noisy state tracking, spatial mapping, and continuous control:

T\-Maze\.The agent receives a cue at the start of a corridor indicating which direction to turn at the far end, but the cue is absent from the observation thereafter\. It must carry a single bit across the entire corridor, isolating long\-horizon recall\. We use a corridor of length 75 \(tmaze\_75\) to make it a true test of long term memory capability, compared to the original benchmark’s 10\.

RockSample\.A rover on ann×nn\\times ngrid must sample good rocks and avoid bad ones, using a noisy sensor whose accuracy decays with distance\. Solving it requires integrating repeated noisy checks to track the latent quality of each rock\. We evaluate the\(11,11\)\(11,11\)and\(15,15\)\(15,15\)grid size configurations\.

Battleship\.The agent fires on a10×1010\\times 10board with hidden ships and observes only a hit or miss each step, without past outcomes retained in the observation\. It must build and maintain an internal map of where it has already fired and what has been hit\.

Masked Walker and HalfCheetah\.Continuous\-control locomotion tasks \(Brax\) in which only velocity features are observed and all positional state is masked\. The agent must integrate the velocity history to recover position, testing memory in a high\-dimensional continuous control setting\.

Craftax \(No Inventory\)\.A pixel\-based open\-world survival task originally proposed in[Hafner \(2021\)](https://arxiv.org/html/2609.38598#bib.bib25)and made harder by[Matthews et al\. \(2024\)](https://arxiv.org/html/2609.38598#bib.bib53)\. The version we run is a futher modified version by[Tao et al\. \(2025\)](https://arxiv.org/html/2609.38598#bib.bib24), in which the inventory panel is cropped from the observation\. The agent must remember its collected resources and progress through the achievement tree from visual input alone, requiring higher memory capabilities than the original Craftax task\. We downsample the observation image size, just as described in the POBAX benchmark\.

### 5\.2Experimental Setup

Our implementations of the unitary recurrent networks are built on top of the POBAX benchmark\([Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\), written entirely in JAX\. We compare against two baselines from POBAX: the GRU recurrent PPO agent, and PPO\-LD, which augments recurrent PPO with theλ\\lambda\-discrepancy memory objective based on a second value function approximator\([Allen et al\., 2024](https://arxiv.org/html/2609.38598#bib.bib54)\)\. All architectural details apart from the RNN, including last action concatenation are kept here\. Each uRNN variant is a drop\-in replacement of the GRU cell, leaving the encoder, heads, and PPO objective untouched\. We evaluate the three unitary parameterizations introduced in Section[4](https://arxiv.org/html/2609.38598#S4):

1. 1\.uRNN: original parameterization with a constant transitionUU\(Section[4\.1](https://arxiv.org/html/2609.38598#S4.SS1)\)\.
2. 2\.EUNN: tunable parameterization with capacityL=2L=2\(Section[4\.2](https://arxiv.org/html/2609.38598#S4.SS2)\)\.
3. 3\.ICuRNN: input\-conditioned transitionU⁡\(xt\)U\(x\_\{t\}\)\(Section[4\.3](https://arxiv.org/html/2609.38598#S4.SS3)\)\.

All uRNN variants use a complex hidden state matching the GRU baseline’s hidden size, and use the same encoder, policy and value heads\. For the EUNN variant, we only evaluate the simplest tuning setting, withL=2L=2, as an initial demonstration given the original work\([Jing et al\., 2017b](https://arxiv.org/html/2609.38598#bib.bib22)\)shows that this setting is already comparable with traditional recurrent cells\. We leave the investigation of the effects of tuning to future work\. All evaluations are done for the same total steps as in the POBAX benchmark, averaged over 5 seeds, except for craftax, where we limit the runs to 100M steps\.

#### Hyperparameters

We evaluate the baselines GRU and PPO\-LD on the same tuned hyperparameter settings in POBAX\. We observe that the complex valued parameters in our methods require a much smaller and separate learning rate than the real valued ones\. We keep the network sizes, hidden state sizes, discount factor and number of parallel environments identical to the baselines, but find that a larger number of steps \(256\) per PPO update works well for the uRNNs\. The optimal GAE lambda, learning rates and entropy coefficient were found through sweeps over a grid, and the details per environment can be found in App\.[E](https://arxiv.org/html/2609.38598#A5)\. We do not tune the settings for the No Inventory Craftax environment, given its large number of parallel environments and compute constraints\.

![Refer to caption](https://arxiv.org/html/2609.38598v1/reward_grid_unitary_best256.png)Figure 2:Reward curves for the 3 uRNN variants and the baselines across 7 environments from the POBAX\([Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\)benchmark

### 5\.3Results

The results of the evaluations across the 7 tasks on the 3 unitary recurrent variants are visualized in Fig\.[2](https://arxiv.org/html/2609.38598#S5.F2)\. Across the suite, the three unitary variants collectively outperform both the GRU and PPO\-LD baselines on a majority of tasks, frequently by a wide margin, and remain competitive on the rest\.

The clearest gains appear on the memory\-heavy discrete tasks\. On both RockSample configurations all three uRNN variants pull well above the baselines, which plateau near a return of1212: on the\(11,11\)\(11,11\)grid the original uRNN reaches the highest returns, roughly2×2\\timesthe GRU and PPO\-LD plateaus, with EUNN and IC\-uRNN following, while on the larger\(15,15\)\(15,15\)grid EUNN fares the best\. The largest gains are on No\-Inventory Craftax run with downsampled observations, where uRNN and EUNN far exceed the two GRU baselines111We follow the same observation cropping and downsampling as described in the POBAX\([Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\)paper for all our evaluations but were unable to replicate their results on the Craftax environment\. IC\-uRNN is the exception here, failing to make progress on this task despite being the strongest variant elsewhere\. We attribute this case to insufficient tuning, given compute budget constraints\.

The continuous\-control and remaining tasks are more even\. On the velocity\-only Walker, IC\-uRNN stands out with a substantial lead over all other methods, suggesting that conditioning the unitary transition on the observation is particularly useful for integrating velocity history, whereas on HalfCheetah all methods perform comparably\. On T\-Maze\-75, the memory\-capable all the methods converge to similar returns, comfortably above the PPO\-LD baseline, with EUNN learning fastest early in training\. Overall, the complex\-recurrent cells deliver consistent and often large improvements on memory\-intensive tasks, with different parameterizations excelling on different environments\.

## 6Phase Aware Policies: A Quantum Information Perspective

In the preceding sections we demonstrated that complex\-valued representations provide a richer source for learning policies in POMDP settings\. However, we believe that naïvely treating uRNN cells as a simple drop\-in replacement for canonical recurrent units leaves untapped potential in the expressivity of the complex\-valued representations\. That is, our approach so far has treated both the real and imaginary parts of the hidden state as real\-valued inputs to the policy head\. However, a complex vector encodes more than just that \- the different dimensions of the vector have different phases and these get lost in the conversion detailed above\. This leads us to ask the question:Can the phases of the hidden state encode relevant information towards the actions an optimal policy should take at each step?In the quest of instilling "phase awareness" into our policies, we draw a parallel to concepts from Quantum Information Theory\.

Quantum states are primarily represented vectors in a complex eigen basis, that is, they can be expressed as a linear combination of a set of orthogonal eigenvectors\. A ’measurement’ in this basis,collapsesthe state onto one of the eigenvectors, with a probability determined by the Born rule\([Born, 1984](https://arxiv.org/html/2609.38598#bib.bib59)\)\. A introduction to these concepts and notation used below is included in Appendix[D](https://arxiv.org/html/2609.38598#A4)\.

This probabilistic nature of measurement, so fundamental to physical quantities and phenomena, sparks a comparison to the policy we are learning here\. Consider a stochastic policyπ⁡\(at∣st\)\\pi\(a\_\{t\}\\mid s\_\{t\}\)over adiscreteaction space𝒜\\mathcal\{A\}\. This policy at any time stepttcan be expressed as a quantum state vector:

\|πt⟩=∑a∈𝒜cta​\|a⟩,cta∈ℂ\\ket\{\\pi\_\{t\}\}=\\sum\_\{a\\in\\mathcal\{A\}\}c^\{a\}\_\{t\}\\ket\{a\},\\quad c^\{a\}\_\{t\}\\in\\mathbb\{C\}\(7\)
Drawing on the Born rule \(Appendix[D](https://arxiv.org/html/2609.38598#A4)\), the probability of observing actionaais:

π⁡\(at∣st\)=\|⟨a\|πt⟩\|2=\|cta\|2\\pi\(a\_\{t\}\\mid s\_\{t\}\)=\|\\braket\{a\|\\pi\_\{t\}\}\|^\{2\}=\|c^\{a\}\_\{t\}\|^\{2\}\(8\)
Given the complex hidden stateht∈ℂnh\_\{t\}\\in\\mathbb\{C\}^\{n\}from the uRNN, we propose learning the action coefficients directly:

\|π⁡\(st\)⟩=∑a∈𝒜c^θa​\(ht\)​\|a⟩,c^θa​\(ht\)∈ℂ,ht∈ℂn\\ket\{\\pi\(s\_\{t\}\)\}=\\sum\_\{a\\in\\mathcal\{A\}\}\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\\ket\{a\},\\quad\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\\in\\mathbb\{C\},\\;h\_\{t\}\\in\\mathbb\{C\}^\{n\}\(9\)
To this end, we replace the real\-valued policy head with a complex\-valued one with the ModReLU activation\. The action coefficientsc^θa​\(ht\)\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)are unnormalized\. To apply the Born rule while maintaining compatibility with softmax, we outputlog⁡\(\|c^θa​\(ht\)\|2\)\\log\(\|\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\|^\{2\}\)as logits:

π⁡\(at∣st\)=exp⁡\(l​o​g​\(\|c^θa​\(ht\)\|2\)\)∑a∈𝒜exp⁡\(l​o​g​\(\|c^θa​\(ht\)\|2\)\)=\|c^θa​\(ht\)\|2∑a∈𝒜\|c^θa​\(ht\)\|2\\pi\(a\_\{t\}\\mid s\_\{t\}\)=\\frac\{\\exp\(log\(\|\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\|^\{2\}\)\)\}\{\\sum\_\{a\\in\\mathcal\{A\}\}\\exp\(log\(\|\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\|^\{2\}\)\)\}=\\frac\{\|\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\|^\{2\}\}\{\\sum\_\{a\\in\\mathcal\{A\}\}\|\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\|^\{2\}\}\(10\)
#### Interference in fully complex policies

Because the action coefficients are a phase\-preserving transformations \(ModReLU preserves phase\) of the hidden state, for actionaa:

log⁡\(\|c^θa​\(ht\)\|2\)=log⁡\(\|wa​\[0\]​ht​\[0\]\+wa​\[1\]​ht​\[1\]\+⋯\+wa​\[n−1\]​ht​\[n−1\]\|2\)\\log\\left\(\|\\hat\{c\}^\{a\}\_\{\\theta\}\(h\_\{t\}\)\|^\{2\}\\right\)=\\log\\left\(\|w^\{a\}\[0\]h\_\{t\}\[0\]\+w^\{a\}\[1\]h\_\{t\}\[1\]\+\\cdots\+w^\{a\}\[n\-1\]h\_\{t\}\[n\-1\]\|^\{2\}\\right\)\(11\)wherewa∈ℂnw^\{a\}\\in\\mathbb\{C\}^\{n\}is the weight vector for actionaa\. Since both weights and hidden states are complex, the phases interfere constructively or destructively \(Equation \([12](https://arxiv.org/html/2609.38598#A2.E12)\)\)\. Because the hidden state contains information from past time steps and the current observation, this interplay allows for potentially a much richer representation space for the policy\.

#### Unitarity

Furthermore, the evolution of quantum states through time, are also constrained by Unitary transformations\. While our recurrent dynamics are not exactly unitary, owing to the addition of the observation dependent vector, this connection between concepts of reinforcement learning and quantum information theory is noteworthy\.

### 6\.1Experiments: Phase Aware Policy uRNNs

Following the methodology described above, we integrate the fully complex policy head with the Born rule based sampling into the 3 previously specified uRNN variants, to have: QuRNN, QICuRNN and QEUNN\. As we described the method for discrete action spaces, we limit our evaluation of the QuRNN variants to exclude the Walker and HalfCheetah environments\. All other settings and hyperparameters \(Appendix[E](https://arxiv.org/html/2609.38598#A5)\) stay the same as Section[5\.2](https://arxiv.org/html/2609.38598#S5.SS2)\.

![Refer to caption](https://arxiv.org/html/2609.38598v1/born_reward_grid_best256.png)Figure 3:Reward curves for the 3 Phase Aware QuRNN variants plotted with a GRU baseline and the best uRNN method on 5 environments with discrete action spaces#### Results\.

The reward curves for the three phase\-aware variants are shown in Figure[3](https://arxiv.org/html/2609.38598#S6.F3), alongside the GRU baseline and the best base uRNN method from Section[5](https://arxiv.org/html/2609.38598#S5)for clarity\. Overall, the phase\-aware policies are competitive with their base uRNN counterparts on most tasks\. The performance is strongest again on No\-Inventory Craftax, where QuRNN and QuEUNN reach close to5×5\\timesthe GRU return and track the best base uRNN, and on RockSample, where QIC\-uRNN slightly exceeds the best base uRNN variant\. On T\-Maze\-75 and Battleship evironments, all methods end up tightly clustered around the GRU baseline\. The exception mirrors the base results: QIC\-uRNN fails to learn on Craftax, just as IC\-uRNN did, while remaining one of the stronger variants on RockSample\. These results indicate that preserving the phase through to action sampling retains the benefits of the complex\-valued recurrence without sacrificing performance relative to the real\-valued policy heads\.

## 7Discussion and Conclusion

In this work, we revisited unitary evolution recurrent neural networks as a representation for reinforcement learning in partially observable environments\. We integrated their complex\-valued, norm\-preserving recurrence into recurrent PPO as a drop\-in replacement for the GRU cell, and studied three parameterizations of the unitary transition\. We then proposed a phase\-aware policy head that samples actions through a Born\-rule mechanism, preserving the complex representation through to action selection\. The strong performance of our methods across the POBAX benchmark are evidence that complex\-valued representations and unitary dynamics are a promising new direction for representation learning in RL, rather than a niche architectural choice\. The connection to quantum information theory further hints at a framework bridging the two fields, as previously explored in a different context by[Dong et al\. \(2008\)](https://arxiv.org/html/2609.38598#bib.bib21)\. We believe these findings motivate a deeper investigation of complex\-valued states for memory and partial observability in RL

Limitations\.We do not explore the hyperparameter space in depth\. Given the known sensitivity of PPO to hyperparameters\([Eimer et al\., 2023](https://arxiv.org/html/2609.38598#bib.bib23);[Huang et al\., 2022](https://arxiv.org/html/2609.38598#bib.bib9)\), more thorough exploration is needed\. Our experiments are limited to part of the POBAX benchmark; evaluation on benchmarks like Memory Gym\([Pleines et al\., 2023](https://arxiv.org/html/2609.38598#bib.bib13)\)and Memory Maze\([Pasukonis et al\., 2022](https://arxiv.org/html/2609.38598#bib.bib39)\)would strengthen the argument\. While we demonstrate the performance of the base version of the EUNN architecture\([Jing et al\., 2017b](https://arxiv.org/html/2609.38598#bib.bib22)\), we only evaluate the simplest tuning setting as a preliminary set of experiments, and we hope to understand the impact of the tuning parameter, and the performance of the method when accessing larger unitary spaces in the future\.

Future Work\.Looking to the future, we hope to extend this work in two main directions\. The dual representation of the complex hidden state naturally motivates an exciting question:what do the phases of the hidden states encode throughout the rollout?We hope to investigate this mechanistically in upcoming work\. A natural extension is to apply complex\-valued representations and unitary transformations to model\-based RL, where the norm\-preserving property becomes central to state transitions in learned world models\. More diverse environments and deeper analysis of training dynamics are also important directions\.

## Acknowledgments

André Biedenkapp is funded by the Deutsche Forschungsgemeinschaft \(DFG, German Research Foundation\) – 572775489\. The authors acknowledge support by the state of Baden\-Württemberg through bwHPC and the German Research Foundation \(DFG\) through grant INST 35/1597\-1 FUGG\. Efstratios Gavves and Sathya Kamesh Bhethanabhotla are supported by the European Union’s Horizon Europe research and innovation programme under grant agreement number 101214398 \(ELLIOT\)\.

## References

- Abadiet al\.\(2015\)M\. Abadi, A\. Agarwal, P\. Barham, E\. Brevdo, Z\. Chen, C\. Citro, G\. S\. Corrado, A\. Davis, J\. Dean, M\. Devin, S\. Ghemawat, I\. Goodfellow, A\. Harp, G\. Irving, M\. Isard, Y\. Jia, R\. Jozefowicz, L\. Kaiser, M\. Kudlur, J\. Levenberg, D\. Mané, R\. Monga, S\. Moore, D\. Murray, C\. Olah, M\. Schuster, J\. Shlens, B\. Steiner, I\. Sutskever, K\. Talwar, P\. Tucker, V\. Vanhoucke, V\. Vasudevan, F\. Viégas, O\. Vinyals, P\. Warden, M\. Wattenberg, M\. Wicke, Y\. Yu, and X\. ZhengTensorFlow: large\-scale machine learning on heterogeneous systems\.Note:Software available from tensorflow\.orgExternal Links:[Link](https://www.tensorflow.org/)Cited by:[Appendix A](https://arxiv.org/html/2609.38598#A1.p1.1)\.
- Allenet al\.\(2024\)C\. Allen, A\. Kirtland, R\. Y\. Tao, S\. Lobel, D\. Scott, N\. Petrocelli, O\. Gottesman, R\. Parr, M\. L\. Littman, and G\. KonidarisMitigating partial observability in sequential decision processes via the lambda discrepancy\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2609.38598#S5.SS2.p1.1)\.
- Aminet al\.\(2011\)Md\. F\. Amin, M\. I\. Amin, A\. Y\. H\. Al\-Nuaimi, and K\. MuraseWirtinger calculus based gradient descent and levenberg\-marquardt learning algorithms in complex\-valued neural networks\.InNeural Information Processing,B\. Lu, L\. Zhang, and J\. Kwok \(Eds\.\),pp\. 550–559\.Cited by:[Appendix A](https://arxiv.org/html/2609.38598#A1.p2.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px3.p1.1)\.
- Arjovskyet al\.\(2016\)M\. Arjovsky, A\. Shah, and Y\. BengioUnitary evolution recurrent neural networks\.InProceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19\-24, 2016,M\. Balcan and K\. Q\. Weinberger \(Eds\.\),JMLR Workshop and Conference Proceedings\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.38598#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2609.38598#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.38598#S4.SS1.p2.1),[Abstract](https://arxiv.org/html/2609.38598#abstract1.1)\.
- Åström, Karl Johan \(1965\)Åström, Karl JohanOptimal Control of Markov Processes with Incomplete State Information I\.Journal of Mathematical Analysis and Applications10\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.38598#S2.SS1.p1.1)\.
- Basseyet al\.\(2021\)J\. Bassey, L\. Qian, and X\. LiA survey of complex\-valued neural networks\.External Links:2101\.12249Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px3.p1.1)\.
- Bellemareet al\.\(2013\)M\. G\. Bellemare, Y\. Naddaf, J\. Veness, and M\. BowlingThe arcade learning environment: an evaluation platform for general agents\.Journal of Artificial Intelligence Research47,pp\. 253–279\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.38598#S2.SS1.p3.1)\.
- Bengioet al\.\(1994\)Y\. Bengio, P\. Simard, and P\. FrasconiLearning long\-term dependencies with gradient descent is difficult\.IEEE Transactions on Neural Networks5\.Cited by:[§2\.2](https://arxiv.org/html/2609.38598#S2.SS2.p1.1)\.
- Born \(1984\)M\. BornZur quantenmechanik der stoßvorgänge \(1926\)\.InDie Deutungen der Quantentheorie,pp\. 48–52\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p4.1),[§6](https://arxiv.org/html/2609.38598#S6.p2.1)\.
- Bradburyet al\.\(2018\)JAX: composable transformations of Python\+NumPy programsExternal Links:[Link](http://github.com/jax-ml/jax)Cited by:[Appendix A](https://arxiv.org/html/2609.38598#A1.p2.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p2.1)\.
- Chenet al\.\(2021\)L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. MordatchDecision transformer: Reinforcement learning via sequence modeling\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.38598#S2.SS1.p4.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1)\.
- Choet al\.\(2014\)K\. Cho, B\. van Merriënboer, C\. Gulcehre, D\. Bahdanau, F\. Bougares, H\. Schwenk, and Y\. BengioLearning phrase representations using RNN encoder–decoder for statistical machine translation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§4\.1](https://arxiv.org/html/2609.38598#S4.SS1.p1.1)\.
- Chuaet al\.\(2018\)K\. Chua, R\. Calandra, R\. McAllister, and S\. LevineDeep reinforcement learning in a handful of trials using probabilistic dynamics models\.InProceedings of the Annual Conference on Neural Information Processing Systems 2018 \(NeurIPS 2018\),S\. Bengio, H\. M\. Wallach, H\. Larochelle, K\. Grauman, N\. Cesa\-Bianchi, and R\. Garnett \(Eds\.\),pp\. 4759–4770\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Donget al\.\(2008\)D\. Dong, C\. Chen, H\. Li, and T\. TarnQuantum reinforcement learning\.Trans\. Sys\. Man Cyber\. Part B38\(5\)\.Cited by:[§7](https://arxiv.org/html/2609.38598#S7.p1.1)\.
- Eimeret al\.\(2023\)T\. Eimer, M\. Lindauer, and R\. RaileanuHyperparameters in reinforcement learning and how to tune them\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202\.Cited by:[§7](https://arxiv.org/html/2609.38598#S7.p2.1)\.
- Elelimyet al\.\(2024\)E\. Elelimy, A\. White, M\. Bowling, and M\. WhiteReal\-time recurrent learning using trace units in reinforcement learning\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p1.1)\.
- Geuchen and Voigtlaender \(2023\)P\. Geuchen and F\. VoigtlaenderOptimal approximation using complex\-valued neural networks\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px3.p1.1)\.
- Glorot and Bengio \(2010\)X\. Glorot and Y\. BengioUnderstanding the difficulty of training deep feedforward neural networks\.InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,Y\. W\. Teh and M\. Titterington \(Eds\.\),Proceedings of Machine Learning Research, Vol\.9\.Cited by:[§4\.1](https://arxiv.org/html/2609.38598#S4.SS1.SSS0.Px1.p1.1)\.
- Gu and Dao \(2024\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.External Links:2312\.00752Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2609.38598#S4.SS3.p1.1)\.
- Guet al\.\(2021\)A\. Gu, I\. Johnson, K\. Goel, K\. Saab, T\. Dao, A\. Rudra, and C\. RéCombining recurrent, convolutional, and continuous\-time models with linear state space layers\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Ha and Schmidhuber \(2018\)D\. Ha and J\. SchmidhuberRecurrent world models facilitate policy evolution\.InAdvances in Neural Information Processing Systems 31,pp\. 2451–2463\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Haarnojaet al\.\(2017\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Hafneret al\.\(2019\)D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. DavidsonLearning latent dynamics for planning from pixels\.InInternational Conference on Machine Learning,pp\. 2555–2565\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Hafneret al\.\(2025\)D\. Hafner, J\. Pasukonis, J\. Ba, and T\. LillicrapMastering diverse control tasks through world models\.Nature,pp\. 1–7\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Hafner \(2021\)D\. HafnerBenchmarking the spectrum of agent capabilities\.arXiv preprint arXiv:2109\.06780\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2609.38598#S5.SS1.p6.1)\.
- Hansenet al\.\(2024\)N\. Hansen, H\. Su, and X\. WangTD\-mpc2: scalable, robust world models for continuous control\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Hausknecht and Stone \(2015\)M\. J\. Hausknecht and P\. StoneDeep recurrent q\-learning for partially observable mdps\.In2015 AAAI Fall Symposia, Arlington, Virginia, USA, November 12\-14, 2015,Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.38598#S2.SS1.p3.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1)\.
- Helfrichet al\.\(2018\)K\. Helfrich, D\. Willmott, and Q\. YeOrthogonal recurrent neural networks with scaled cayley transform\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p1.1)\.
- Hochreiter and Schmidhuber \(1997\)S\. Hochreiter and J\. SchmidhuberLong short\-term memory\.Neural Comput\.9\(8\)\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.38598#S2.SS2.p1.1)\.
- Huanget al\.\(2022\)S\. Huang, R\. F\. J\. Dossa, A\. Raffin, A\. Kanervisto, and W\. WangThe 37 implementation details of proximal policy optimization\.InICLR Blog Track,External Links:[Link](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)Cited by:[§7](https://arxiv.org/html/2609.38598#S7.p2.1)\.
- Jinget al\.\(2017a\)L\. Jing, C\. Gulcehre, J\. Peurifoy, Y\. Shen, M\. Tegmark, M\. Soljačić, and Y\. BengioGated orthogonal recurrent units: on learning to forget\.External Links:1706\.02761Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p1.1)\.
- Jinget al\.\(2017b\)L\. Jing, Y\. Shen, T\. Dubcek, J\. Peurifoy, S\. Skirlo, Y\. LeCun, M\. Tegmark, and M\. SoljačićTunable efficient unitary neural networks \(EUNN\) and their application to RNNs\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research\.Cited by:[§5\.2](https://arxiv.org/html/2609.38598#S5.SS2.p2.1),[§7](https://arxiv.org/html/2609.38598#S7.p2.1)\.
- Jinget al\.\(2017c\)L\. Jing, Y\. Shen, T\. Dubcek, J\. Peurifoy, S\. Skirlo, Y\. LeCun, M\. Tegmark, and M\. SoljačićTunable efficient unitary neural networks \(EUNN\) and their application to RNNs\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p2.1),[§4\.2](https://arxiv.org/html/2609.38598#S4.SS2.p1.1),[Abstract](https://arxiv.org/html/2609.38598#abstract1.1)\.
- Kaelblinget al\.\(1998\)L\. P\. Kaelbling, M\. L\. Littman, and A\. R\. CassandraPlanning and acting in partially observable stochastic domains\.Artif\. Intell\.101,pp\. 99–134\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Katharopouloset al\.\(2020\)A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. FleuretTransformers are rnns: fast autoregressive transformers with linear attention\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§4\.3](https://arxiv.org/html/2609.38598#S4.SS3.p1.1)\.
- Lambrechtset al\.\(2022\)G\. Lambrechts, A\. Bolland, and D\. ErnstRecurrent networks, hidden states and beliefs in partially observable environments\.Transactions on Machiene Learning Research\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Matthewset al\.\(2024\)M\. Matthews, M\. Beukman, B\. Ellis, M\. Samvelyan, M\. Jackson, S\. Coward, and J\. FoersterCraftax: a lightning\-fast benchmark for open\-ended reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§5\.1](https://arxiv.org/html/2609.38598#S5.SS1.p6.1)\.
- Mhammediet al\.\(2017\)Z\. Mhammedi, A\. D\. Hellicar, A\. Rahman, and J\. BaileyEfficient orthogonal parametrisation of recurrent neural networks using householder reflections\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p1.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski, S\. Petersen, C\. Beattie, A\. Sadik, I\. Antonoglou, H\. King, D\. Kumaran, D\. Wierstra, S\. Legg, and D\. HassabisHuman\-level control through deep reinforcement learning\.Nature518\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.38598#S2.SS1.p3.1)\.
- Moradet al\.\(2023\)S\. Morad, R\. Kortvelesy, M\. Bettini, S\. Liwicki, and A\. ProrokPOPGym: benchmarking partially observable reinforcement learning\.InThe Eleventh International Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1)\.
- Niet al\.\(2022\)T\. Ni, B\. Eysenbach, and R\. SalakhutdinovRecurrent model\-free RL can be a strong baseline for many pomdps\.InProceedings of the International Conference on Machine Learning, ICML 2022,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvári, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research,pp\. 16691–16723\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Niet al\.\(2023\)T\. Ni, M\. Ma, B\. Eysenbach, and P\. BaconWhen do transformers shine in RL? Decoupling memory from credit assignment\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1)\.
- Nielsen and Chuang \(2010\)M\. A\. Nielsen and I\. L\. ChuangQuantum computation and quantum information: 10th anniversary edition\.Cambridge University Press,Cambridge\.Cited by:[Appendix D](https://arxiv.org/html/2609.38598#A4.p1.1)\.
- Orvietoet al\.\(2023\)A\. Orvieto, S\. L\. Smith, A\. Gu, A\. Fernando, C\. Gulcehre, R\. Pascanu, and S\. DeResurrecting recurrent neural networks for long sequences\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p1.1)\.
- Parisottoet al\.\(2020\)E\. Parisotto, F\. Song, J\. Rae, R\. Pascanu, C\. Gulcehre, S\. Jayakumar, M\. Jaderberg, R\. L\. Kaufman, A\. Clark, S\. Noury, M\. Botvinick, N\. Heess, and R\. HadsellStabilizing transformers for reinforcement learning\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119\.Cited by:[§2\.1](https://arxiv.org/html/2609.38598#S2.SS1.p4.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1)\.
- Pasukoniset al\.\(2022\)J\. Pasukonis, T\. Lillicrap, and D\. HafnerEvaluating long\-term memory in 3d mazes\.arXiv preprint arXiv:2210\.13383\.Cited by:[§7](https://arxiv.org/html/2609.38598#S7.p2.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. ChintalaPyTorch: an imperative style, high\-performance deep learning library\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Cited by:[Appendix A](https://arxiv.org/html/2609.38598#A1.p1.1)\.
- Pleineset al\.\(2023\)M\. Pleines, M\. Pallasch, F\. Zimmer, and M\. PreussMemory gym: partially observable challenges to memory\-based agents\.InThe Eleventh International Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.38598#S7.p2.1)\.
- Samsamiet al\.\(2024\)M\. R\. Samsami, A\. Zholus, J\. Rajendran, and S\. ChandarMastering memory tasks with world models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px2.p1.1)\.
- Sarroffet al\.\(2015\)A\. M\. Sarroff, V\. Shepardson, and M\. A\. CaseyLearning representations using complex\-valued nets\.External Links:1511\.06351Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px3.p1.1)\.
- Schulmanet al\.\(2017a\)J\. Schulman, S\. Levine, P\. Moritz, M\. I\. Jordan, and P\. AbbeelTrust region policy optimization\.External Links:1502\.05477Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Schulmanet al\.\(2017b\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.External Links:1707\.06347Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Siemset al\.\(2025\)J\. Siems, T\. Carstensen, A\. Zela, F\. Hutter, M\. Pontil, and R\. GrazziDeltaProduct: improving state\-tracking in linear rnns via householder products\.External Links:2502\.10297Cited by:[§4\.3](https://arxiv.org/html/2609.38598#S4.SS3.p1.1)\.
- Smirnov and Gu \(2025\)I\. Smirnov and S\. GuRLBenchNet: the right network for the right reinforcement learning task\.External Links:2505\.15040Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p1.1)\.
- Sutton and Barto \(1998\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.MIT Press,Cambridge, MA\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p1.1)\.
- Taoet al\.\(2025\)R\. Y\. Tao, K\. Guo, C\. Allen, and G\. KonidarisBenchmarking partial observability in reinforcement learning with a suite of memory\-improvable domains\.The Reinforcement Learning Journal\.Cited by:[Table 2](https://arxiv.org/html/2609.38598#A5.T2),[Table 3](https://arxiv.org/html/2609.38598#A5.T3),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2609.38598#S4.SS1.p1.1),[Figure 2](https://arxiv.org/html/2609.38598#S5.F2),[§5\.1](https://arxiv.org/html/2609.38598#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2609.38598#S5.SS1.p6.1),[§5\.2](https://arxiv.org/html/2609.38598#S5.SS2.p1.1),[footnote 1](https://arxiv.org/html/2609.38598#footnote1)\.
- Trabelsiet al\.\(2018\)C\. Trabelsi, O\. Bilaniuk, Y\. Zhang, D\. Serdyuk, S\. Subramanian, J\. F\. Santos, S\. Mehri, N\. Rostamzadeh, Y\. Bengio, and C\. J\. PalDeep complex networks\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.38598#A1.p1.1),[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px3.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2609.38598#S1.p2.1)\.
- Wisdomet al\.\(2016\)S\. Wisdom, T\. Powers, J\. Hershey, J\. Le Roux, and L\. AtlasFull\-capacity unitary recurrent neural networks\.Vol\.29\.Cited by:[§3](https://arxiv.org/html/2609.38598#S3.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2609.38598#S4.SS2.p1.1)\.
- Yanget al\.\(2025\)S\. Yang, J\. Kautz, and A\. HatamizadehGated delta networks: improving mamba2 with delta rule\.InProceedings of ICLR,Cited by:[§4\.3](https://arxiv.org/html/2609.38598#S4.SS3.p1.1)\.
- Yanget al\.\(2024\)S\. Yang, B\. Wang, Y\. Zhang, Y\. Shen, and Y\. KimParallelizing linear transformers with the delta rule over sequence length\.InProceedings of NeurIPS,Cited by:[§4\.3](https://arxiv.org/html/2609.38598#S4.SS3.p1.1)\.

## Appendix AComplex Forward and Backward Passes

When uRNNs were first proposed, PyTorch\[[Paszke et al\., 2019](https://arxiv.org/html/2609.38598#bib.bib3)\]and TensorFlow\[[Abadi et al\., 2015](https://arxiv.org/html/2609.38598#bib.bib2)\]did not offer reliable complex data types, so researchers had to write custom layers for initialization, normalization, and backpropagation\. Works like\[[Trabelsi et al\., 2018](https://arxiv.org/html/2609.38598#bib.bib29)\]still existed to describe the mechanics of complex\-valued neural networks, defined activations, layers and normalization techniques around them, while demonstrating their applicability in various domains\.

Today the situation is much simpler\. Both PyTorch and JAX\[[Bradbury et al\., 2018](https://arxiv.org/html/2609.38598#bib.bib5)\]include built\-in complex dtypes \(for exampletorch\.complex64andjax\.numpy\.complex64\) and their autograd systems support Wirtinger calculus automatically\[[Amin et al\., 2011](https://arxiv.org/html/2609.38598#bib.bib46)\]\. This means a complex forward or backward pass can be written just like its real\-valued counterpart: declare parameters in a complex dtype and the framework handles gradient propagation\.

In this work, we rely heavily on these frameworks for a much simpler and straightforward implementation of complex\-valued neural networks\.

## Appendix BComplex Algebra and Unitary Matrices

An important idea in this work is the extension of linear algebra to the complex space\. Complex numbers and complex\-valued representations provide an additional degree of freedom in the form of a phase\.

A complex numberz=x\+i​yz=x\+iycan also be represented asz=r​ei​θz=re^\{i\\theta\}withr=\|z\|r=\|z\|andθ=tan−1⁡\(y/x\)\\theta=\\tan^\{\-1\}\(y/x\)\. The product of two complex numbers can be seen as rotations in phase space:z1​z2=r1​r2​ei⁡\(θ1\+θ2\)z\_\{1\}z\_\{2\}=r\_\{1\}r\_\{2\}e^\{i\(\\theta\_\{1\}\+\\theta\_\{2\}\)\}\.

An important interaction is the squared norm of the sum of two complex numbers:

\|z1\+z2\|2=\|z1\|2\+\|z2\|2\+2​\|z1\|​\|z2\|​cos⁡\(θ2−θ1\)\|z\_\{1\}\+z\_\{2\}\|^\{2\}=\|z\_\{1\}\|^\{2\}\+\|z\_\{2\}\|^\{2\}\+2\|z\_\{1\}\|\|z\_\{2\}\|\\cos\(\\theta\_\{2\}\-\\theta\_\{1\}\)\(12\)This shows that the squared norm depends on the phase difference, a property we will exploit later\.

A complex vector spaceℂn\\mathbb\{C\}^\{n\}is a Hilbert space where the inner product takes the form⟨a,b⟩=a∗​b\\langle a,b\\rangle=a^\{\\ast\}b, with\(⋅\)∗\(\\cdot\)^\{\\ast\}denoting conjugate transpose\.

Now for matrices in a Hilbert space, the unitary property is defined as follows:

###### Definition B\.1\.

A matrixU∈ℂn×nU\\in\\mathbb\{C\}^\{n\\times n\}is*unitary*ifU†​U=IU^\{\\dagger\}U=I, whereU†U^\{\\dagger\}denotes the conjugate transpose\. For any vectorx∈ℂnx\\in\\mathbb\{C\}^\{n\}:‖U​x‖2=‖x‖2\\\|Ux\\\|\_\{2\}=\\\|x\\\|\_\{2\}\.

All eigenvalues of a unitary matrix lie on the complex unit circle \(\|λ\|=1\|\\lambda\|=1\)\.

#### Householder Reflections\.

A Householder reflection \(Fig\.[4](https://arxiv.org/html/2609.38598#A3.F4)\), characterized by a matrix in complex space isH=I−2​v​v∗/\(v∗​v\)H=I\-2\\,vv^\{\\ast\}/\(v^\{\\ast\}v\),v∈ℂnv\\in\\mathbb\{C\}^\{n\}\. All Householder matrices are unitary \(see Appendix[C](https://arxiv.org/html/2609.38598#A3)\), and composing multiple reflections generates a rich family of unitary transformations efficiently\. This idea forms the core of how we can parameterize unitary recurrent systems\.

## Appendix CProof for the unitarity of Householder Matrices

vv\(v∗​x\)​v\(v^\{\*\}x\)vx−\(v∗​x\)​vx\-\(v^\{\*\}x\)vxx−\(v∗​x\)​v\-\(v^\{\*\}x\)v−\(v∗​x\)​v\-\(v^\{\*\}x\)v\(I−2​v​v∗\)​x\(I\-2vv^\{\*\}\)xFigure 4:Geometric interpretation of a Householder reflection\.A vectorxxis reflected across the hyperplane orthogonal tovvto produceH​xHx, whereH=I−2​v​v∗/\(v∗​v\)H=I\-2vv^\{\\ast\}/\(v^\{\\ast\}v\)is the Householder matrix\. The reflection flips the parallel component while preserving the perpendicular component\.In[B](https://arxiv.org/html/2609.38598#A2.SS0.SSS0.Px1), we introduced the Householder reflection as a way to parametrize unitary matrices\. We now provide the proof for the unitarity of Householder matrices\.

###### Theorem C\.1\.

Letv∈ℂnv\\in\\mathbb\{C\}^\{n\}be a nonzero vector\. The Householder matrix defined by

H=I−2​v​v†v†​vH=I\-\\frac\{2vv^\{\\dagger\}\}\{v^\{\\dagger\}v\}is unitary, i\.e\.,H†​H=IH^\{\\dagger\}H=I\.

###### Proof\.

We verify thatH†​H=IH^\{\\dagger\}H=Iby direct computation\.

First, we compute the Hermitian conjugate ofHH:

H†\\displaystyle H^\{\\dagger\}=\(I−2​v​v∗v∗​v\)†\\displaystyle=\\left\(I\-\\frac\{2vv^\{\*\}\}\{v^\{\*\}v\}\\right\)^\{\\dagger\}=I†−2v∗​v​\(v​v∗\)†\\displaystyle=I^\{\\dagger\}\-\\frac\{2\}\{v^\{\*\}v\}\\left\(vv^\{\*\}\\right\)^\{\\dagger\}=I−2\(v∗​v\)∗​\(v∗\)∗​v∗\\displaystyle=I\-\\frac\{2\}\{\(v^\{\*\}v\)^\{\*\}\}\\left\(v^\{\*\}\\right\)^\{\*\}v^\{\*\}=I−2v∗​v​v​v∗\\displaystyle=I\-\\frac\{2\}\{v^\{\*\}v\}vv^\{\*\}=H,\\displaystyle=H,where we used the fact thatv∗​vv^\{\*\}vis real and positive \(hence\(v∗​v\)∗=v∗​v\(v^\{\*\}v\)^\{\*\}=v^\{\*\}v\), and\(v∗\)∗=v\\left\(v^\{\*\}\\right\)^\{\*\}=v\.

Thus,HHis Hermitian\. Now we verify unitarity:

H†​H\\displaystyle H^\{\\dagger\}H=H2\\displaystyle=H^\{2\}=\(I−2​v​v∗v∗​v\)​\(I−2​v​v∗v∗​v\)\\displaystyle=\\left\(I\-\\frac\{2vv^\{\*\}\}\{v^\{\*\}v\}\\right\)\\left\(I\-\\frac\{2vv^\{\*\}\}\{v^\{\*\}v\}\\right\)=I−2​v​v∗v∗​v−2​v​v∗v∗​v\+4​v​v∗​v​v∗\(v∗​v\)2\\displaystyle=I\-\\frac\{2vv^\{\*\}\}\{v^\{\*\}v\}\-\\frac\{2vv^\{\*\}\}\{v^\{\*\}v\}\+\\frac\{4vv^\{\*\}vv^\{\*\}\}\{\(v^\{\*\}v\)^\{2\}\}=I−4​v​v∗v∗​v\+4​v​\(v∗​v\)​v∗\(v∗​v\)2\\displaystyle=I\-\\frac\{4vv^\{\*\}\}\{v^\{\*\}v\}\+\\frac\{4v\(v^\{\*\}v\)v^\{\*\}\}\{\(v^\{\*\}v\)^\{2\}\}=I−4​v​v∗v∗​v\+4​v​v∗v∗​v\\displaystyle=I\-\\frac\{4vv^\{\*\}\}\{v^\{\*\}v\}\+\\frac\{4vv^\{\*\}\}\{v^\{\*\}v\}=I,\\displaystyle=I,where in the second\-to\-last line we used the fact thatv∗​vv^\{\*\}vis a scalar, sov⁡\(v∗​v\)=\(v∗​v\)​vv\(v^\{\*\}v\)=\(v^\{\*\}v\)v\.

Therefore,H†​H=IH^\{\\dagger\}H=I, which proves thatHHis unitary\. ∎

## Appendix DQuantum Information Theory and Measurements

This section serves as a brief introduction to the concepts of quantum information theory and measurements, which we will use to motivate a alternative perspective from which this work and the underlying Unitary Recurrence can be viewed, in Section[6](https://arxiv.org/html/2609.38598#S6)\. While this section only covers the concepts and relationships relevant to this work, a more comprehensive treatment of the complex linear algebra and quantum mechanical principles can be found in\[[Nielsen and Chuang, 2010](https://arxiv.org/html/2609.38598#bib.bib40)\]\.

Throughout this section, we use Dirac notation: a ket\|ψ⟩\\ket\{\\psi\}denotes a column vector, while a bra⟨ϕ\|\\bra\{\\phi\}denotes a row vector\. A braket⟨ϕ,ψ⟩\\braket\{\\phi,\\psi\}denotes the inner product⟨ϕ\|​\|ψ⟩\\bra\{\\phi\}\\ket\{\\psi\}, and a ketbra\|ψ⟩​⟨ϕ\|\\ket\{\\psi\}\\bra\{\\phi\}denotes the outer product\.

A quantum state provides a complete description of an isolated physical system\.

###### Definition D\.1\.

In a finite\-dimensional Hilbert spaceℋ≅ℂd\\mathcal\{H\}\\cong\\mathbb\{C\}^\{d\}, any pure state is represented as a unit vector

\|ψ⟩=∑k=1dck​\|k⟩,\\ket\{\\psi\}=\\sum\_\{k=1\}^\{d\}c\_\{k\}\\ket\{k\},where\{\|k⟩\}\\\{\\ket\{k\}\\\}is an orthonormal basis and the coefficientsck∈ℂc\_\{k\}\\in\\mathbb\{C\}satisfy∑k\|ck\|2=1\\sum\_\{k\}\|c\_\{k\}\|^\{2\}=1\.

The definition of basis is the same as in real vector spaces, where any set of orthogonal vectors that span the space can be a basis\. This state vector expansion is not merely a coordinate representation: it encodes all physically accessible information about the system\.

A state vector written in this form as a linear combination of the basis eigenstates, is said to be in a*superposition*of the eigenstates\. The coefficientsckc\_\{k\}are the*amplitudes*of the state vector in the basis eigenstates, and if all the amplitudesckc\_\{k\}are equal, then this superposition is a*equal superposition state*, meaning the state vector is equally likely to be in any of the basis eigenstates\.

### D\.1Operators and State Transformations

In quantum mechanics, transformations of quantum states are represented by*Operators*, which are square matrices acting on the Hilbert space\. An operatorA∈ℂd×dA\\in\\mathbb\{C\}^\{d\\times d\}transforms a state\|ψ⟩\\ket\{\\psi\}toA​\|ψ⟩A\\ket\{\\psi\}, analogous to how a matrix multiplies a vector in linear algebra\. Operators can represent various physical processes: rotations, measurements, or time evolution\.

For a measurement to be physically meaningful, the corresponding operator must be*Hermitian*\(as defined in[B](https://arxiv.org/html/2609.38598#A2)\)\. Hermitian operators have real eigenvalues and form a complete orthonormal eigenbasis, making them suitable for representing physical observables, or quantities that can be measured, such as energy or position\. When we measure an observable represented by a Hermitian operatorMM, we are effectively projecting the state onto one ofMM’s eigenstates\.

### D\.2Measurement and Probabilistic Outcomes

A measurement in the eigenbasis\{\|k⟩\}\\\{\\ket\{k\}\\\}of a Hermitian operator is inherently probabilistic\. Upon measurement, the outcomeλk\\lambda\_\{k\}\(corresponding to eigenstate\|k⟩\\ket\{k\}\) is observed with probability

p⁡\(k\)=\|⟨k,ψ⟩\|2=\|ck\|2,p\(k\)=\|\\braket\{k,\\psi\}\|^\{2\}=\|c\_\{k\}\|^\{2\},as prescribed by the Born rule, and the post\-measurement state collapses to\|k⟩\\ket\{k\}\. The stochasticity arises not from noise but from the fundamental quantum mechanical property of projecting the state onto an eigenbasis of the measured observable\. This projection is a linear operation that transforms the state from a superposition\|ψ⟩=∑kck​\|k⟩\\ket\{\\psi\}=\\sum\_\{k\}c\_\{k\}\\ket\{k\}to a single basis state\|k⟩\\ket\{k\}\.

### D\.3Time Evolution and Unitary Dynamics

The time evolution of a closed quantum system is governed by the Schrödinger equation, which is a linear differential equation that describes how the state of a quantum system changes over time\.

i​ℏ​dd​t​\|ψ⟩=H​\|ψ⟩,i\\hbar\\frac\{d\}\{dt\}\\ket\{\\psi\}=H\\ket\{\\psi\},whereHHis the Hamiltonian, a Hermitian operator encoding the system’s energy structure\. Its formal solution is

\|ψ⁡\(t\)⟩=U⁡\(t\)​\|ψ⁡\(0\)⟩,U⁡\(t\)=e−iℏ​H​t\.\\ket\{\\psi\(t\)\}=U\(t\)\\ket\{\\psi\(0\)\},\\qquad U\(t\)=e^\{\-\\frac\{i\}\{\\hbar\}Ht\}\.whereU⁡\(t\)U\(t\)is the time evolution operator\. BecauseHHis Hermitian, the operatorU⁡\(t\)U\(t\)is unitary, ensuring that the norm of the state is preserved during evolution\. This norm preservation mirrors the dynamical behaviour of unitary recurrent updates used in machine learning: the global “energy” of the representation remains constant, and information is carried forward through rotations in complex space rather than amplification or decay\.

## Appendix EHyperparameters

We organize the hyperparameters into three tables\. Table[1](https://arxiv.org/html/2609.38598#A5.T1)lists the PPO settings that are held fixed across the two baselines \(GRU, PPO\-LD\) and all six uRNN variants\. Table[2](https://arxiv.org/html/2609.38598#A5.T2)lists the per\-environment settings that are fixed across all methods but vary by environment \(hidden size, parallel environments, training budget, and discount\)\. Table[3](https://arxiv.org/html/2609.38598#A5.T3)reports the per\-environment best hyperparameters of the two baselines, found by sweeping over a grid\.

Table 1:Hyperparameters held fixed across all baselines and all six uRNN variants\. The rollout lengthTTis the only PPO setting we change relative to the baselines \(see note\)\.HyperparameterValueNoteOptimizerAdamUpdate epochs4per PPO updateMinibatches per update4PPO clipϵ\\epsilon0\.2Value loss coefficientcvc\_\{v\}0\.5Max gradient norm0\.5global\-norm clippingLearning\-rate anneallinear→0\\to 0applied to the real parameter groupAction concatenationyesprevious actionat−1a\_\{t\-1\}appended tooto\_\{t\}Rollout lengthTT128 / 256128 for baselines, 256 for uRNN variants;Craftax uses 64Seeds5Table 2:Per\-environment settings, fixed across all methods\. The complex hidden state of each uRNN variant has the same dimensionnnas the GRU baseline’s hidden state\. Training budgets follow the POBAX benchmark\[[Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\]; Craftax is capped at10810^\{8\}steps\.Table 3:Per\-environment best hyperparameters for the baselines, selected by a grid sweep, exactly following\[[Tao et al\., 2025](https://arxiv.org/html/2609.38598#bib.bib24)\]### E\.1Per\-environment sweeps for the uRNN variants

We tune four hyperparameters per environment for our six variants: the real and complex learning rates \(lr, complex lr\), the entropy coefficient, and the GAE coefficientλ0\\lambda\_\{0\}\. All remaining settings are shared with the baselines \(Tables[1](https://arxiv.org/html/2609.38598#A5.T1)and[2](https://arxiv.org/html/2609.38598#A5.T2)\); the second GAE coefficientλ1=0\.5\\lambda\_\{1\}=0\.5and theλ\\lambda\-discrepancy weight are not used \(β=0\\beta=0, single critic\)\. For each environment we report the search space \(left\) and the best setting per variant \(right\)\. ForCraftaxwe do not tune the uRNN variants: we reuse the baseline settings of Table[3](https://arxiv.org/html/2609.38598#A5.T3)\.

Table 4:T\-Maze \(corridor 75\)\.Search space \(left\) and best per\-variant settings \(right\)\.
Table 5:RockSample\(11,11\)\(11,11\)\.Search space \(left\) and best per\-variant settings \(right\)\.
Table 6:RockSample\(15,15\)\(15,15\)\.Search space \(left\) and best per\-variant settings \(right\)\.
Table 7:Battleship\(10×10\)\(10\\times 10\)\.Search space \(left\) and best per\-variant settings \(right\)\.
Table 8:Masked Walker\.Search space \(left\) and best per\-variant settings \(right\)\. The Born\-rule \(Q\) variants are defined for discrete action spaces only and are not evaluated here\.
Table 9:Masked HalfCheetah\.Search space \(left\) and best per\-variant settings \(right\)\. The Born\-rule \(Q\) variants are defined for discrete action spaces only and are not evaluated here\.