Categorical Internalisation of Environmental Groupoids for Generalisable POMDP Solving

# Categorical Internalisation of Environmental Groupoids for Generalisable POMDP Solving
Source: [https://arxiv.org/html/2609.27745](https://arxiv.org/html/2609.27745)
###### Abstract

This paper advocates category theory as a practical framework for structuring and improving reinforcement learning in high\-dimensional, partially observable environments\. We model symmetries between environmental states by partitioning the state space into equivalence classes induced by symmetry orbits, and organise each such class as a groupoid with a designated canonical representative\. This allows the agent to share what it learns across many similar environmental states simultaneously, rather than treating every orientation or position as an entirely new problem\. Learning is thus carried out on a symmetry\-reduced state space with each orbit represented once, preserving structure while eliminating redundancy and improving sample efficiency\.

We implement this framework within standard reinforcement learning pipelines and evaluate two different approaches on partially observable benchmarks, demonstrating that orbit\-based partitioning yields consistent performance improvements in environments exhibiting latent symmetry\. Beyond these empirical results, our approach illustrates how categorical structure provides a principled bridge between abstract reinforcement learning formulations and their computational application, thereby establishing a pathway toward more structured and scalable learning systems\.

## 1Introduction

Reinforcement learning \(RL\) has seen widespread success in solving complex sequential decision\-making problems, with myriad applications from board games to robotic control systems\[[36](https://arxiv.org/html/2609.27745#bib.bib25),[29](https://arxiv.org/html/2609.27745#bib.bib28)\]\. In the standard framework, an agent interacts with an environment modelled as a Markov Decision Process \(MDP\), using reward information from its interactions to refine a policy that maximizes a cumulative reward\[[37](https://arxiv.org/html/2609.27745#bib.bib20),[34](https://arxiv.org/html/2609.27745#bib.bib10)\]\. However, RL approaches have faced some challenges resolving real\-world, high\-dimensional environmental complexity, particularly with sample inefficiency and poor generalisation\[[20](https://arxiv.org/html/2609.27745#bib.bib37)\]\.

Category theory holds potential to redefine environmental representations for RL agents, and this paper will utilise it as a foundation to accelerate learning by exploiting environmental symmetries\. From a categorical perspective, an MDP can be understood as a coalgebra for the Giry monad, where transitions represent morphisms between state objects\[[17](https://arxiv.org/html/2609.27745#bib.bib44),[24](https://arxiv.org/html/2609.27745#bib.bib45)\]\. Given a measurable state spaceXXand a set of actionsAA, the system dynamics are given by a measurable mapP:X×A→𝒢⁡\(X\)P:X\\times A\\to\\mathcal\{G\}\(X\), where𝒢⁡\(X\)\\mathcal\{G\}\(X\)denotes the space of probability measures overXX\. In practical applications, agents lack complete information and instead of directly accessingXXthey experience a partially observable problem with data given through high\-dimensional, often\-noisy observations\. Thus, maintaining a robust internal representation which properly captures the valuable information is vital and for an agent to generalise across disparate states, it must learn to map raw observations into a latent space that disentangles the underlying factors of variation\[[4](https://arxiv.org/html/2609.27745#bib.bib41),[22](https://arxiv.org/html/2609.27745#bib.bib4)\]\. Disentanglement allows the agent to treat independent properties of the world such as position, color, or velocity, as distinct coordinates in its internal model, creating stronger understandings about the environment while limiting weak conclusions drawn from few samples\[[35](https://arxiv.org/html/2609.27745#bib.bib14)\]\.

One key pathway to effectively achieve disentanglement is by utilising environmental symmetries, the properties inherently invariant under various transformation groups, such as translations or rotations\[[32](https://arxiv.org/html/2609.27745#bib.bib39)\]\. By incorporating these symmetries into the agent’s architecture, an agent can reduce its hypothesis space to augment sample efficiency and support shared learning across different but symmetrical states\[[12](https://arxiv.org/html/2609.27745#bib.bib24),[40](https://arxiv.org/html/2609.27745#bib.bib13)\]\. In this paper, we proposegroupoidsas the natural mathematical language for internal representations in RL, and we show how to use them to capture the local structure of reversible actions and substructure transitions, as has been done in other fields before\[[48](https://arxiv.org/html/2609.27745#bib.bib2),[42](https://arxiv.org/html/2609.27745#bib.bib1)\]\. This groupoid\-based approach both generalises existing symmetry\-based RL methods and also provides a compositional framework for building complex internal models from simpler geometric primitives\[[3](https://arxiv.org/html/2609.27745#bib.bib51)\]\.

This paper will outline a novel RL framework which uses groupoids for internal representations to facilitate shared learning across states and accelerates learning in partially observable environments\. Our contributions are as follows:categorical reinforcement learning: we show how category theory can be used to better represent environments for RL models;groupoid solver algorithm: we implement a practical algorithm using this categorical internal representation for the first time; andempirical results: we use standard benchmarks to demonstrate the performance of our model against traditional approaches\.

In the following, we first review imperative background information to recontextualise reinforcement learning and symmetry\-based methods using category theory\. We will outline both the mathematical and then computational models for groupoid\-based RL with different canonicalisation methods as well as the setup for the testing environments\. Finally, we will analyse the impact of different approaches using key environmental benchmarks, and discuss the results and their implications\.

## 2Background

In this section, we reformulate standard RL in categorical terms to provide a rigorous basis for our model\. We first establish environmental structure, and then outline categorical definitions of symmetry and disentanglement to create the bedrock for canonicalisation methods which reduce sample complexity\.

### 2\.1Reinforcement Learning

RL agents learn through sequential interactions with an environment to maximize cumulative reward\[[37](https://arxiv.org/html/2609.27745#bib.bib20)\]\. Most such problems are formalised as Markov Decision Processes \(MDPs\)\[[34](https://arxiv.org/html/2609.27745#bib.bib10)\], which can be lifted into a categorical framework to reveal their underlying algebraic structure\.

###### Definition 2\.1\(Categorical Markov Decision Process\)\.

A Markov Decision Process \(MDP\) is a tupleℳ=\(S,A,α,γ\)\\mathcal\{M\}=\(S,A,\\alpha,\\gamma\)in the category of measurable spaces, whereSSis the state space,AAis the action space,

α:S×A→𝒢⁡\(S×ℝ\)\\alpha:S\\times A\\to\\mathcal\{G\}\(S\\times\\mathbb\{R\}\)is a Markov kernel assigning to each state\-action pair a probability measure over successor states and rewards, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. Equivalently, by currying,α\\alphainduces a morphism

α~:S→𝒢​\(S×ℝ\)A,\\tilde\{\\alpha\}:S\\to\\mathcal\{G\}\(S\\times\\mathbb\{R\}\)^\{A\},makingSSa coalgebra for the functorF⁡\(X\)=Meas⁡\(A,𝒢⁡\(X×ℝ\)\)F\(X\)=\\mathrm\{Meas\}\(A,\\mathcal\{G\}\(X\\times\\mathbb\{R\}\)\)\.

The classical transition kernelPPand expected reward functionRRare recovered via marginalisation and integration:

P⁡\(s,a,B\)=α⁡\(s,a\)​\(B×ℝ\),R⁡\(s,a\)=∫S×ℝπ2​𝑑α​\(s,a\),P\(s,a,B\)=\\alpha\(s,a\)\(B\\times\\mathbb\{R\}\),\\quad R\(s,a\)=\\int\_\{S\\times\\mathbb\{R\}\}\\pi\_\{2\}\\,d\\alpha\(s,a\),whereπ2:S×ℝ→ℝ\\pi\_\{2\}:S\\times\\mathbb\{R\}\\to\\mathbb\{R\}is the projection onto the reward component\.

In realistic, partially\-observable scenarios, the agent lacks direct access to the state objectSSand so the problem cannot be represented by a standard MDP\. Partially Observable Markov Decision Processes \(POMDPs\) extend the MDP framework by introducing an observation spaceOOand an observation morphismΩ\\Omegawhich together represent the portion of the environment the agent actually knows about\.

###### Definition 2\.2\(Categorical POMDP\)\.

A Partially Observable Markov Decision Process \(POMDP\) is a tuple𝒫=\(S,A,Z,α,Ω,γ\)\\mathcal\{P\}=\(S,A,Z,\\alpha,\\Omega,\\gamma\), where\(S,A,α,γ\)\(S,A,\\alpha,\\gamma\)forms a categorical MDP,ZZis the observation space, and

Ω:S→𝒢⁡\(Z\)\\Omega:S\\to\\mathcal\{G\}\(Z\)is the observation kernel assigning to each state a probability measure over observations\.

In this framework, the agent’s decision\-making policy cannot in general be implemented as a simple morphismπ:S→𝒢⁡\(A\)\\pi:S\\to\\mathcal\{G\}\(A\), sinceSSis not directly observable\. Instead, the agent maintains a belief stateb∈𝒢⁡\(S\)b\\in\\mathcal\{G\}\(S\), representing a probability distribution over possible states, and acts according to a policyπ:𝒢⁡\(S\)→𝒢⁡\(A\)\\pi:\\mathcal\{G\}\(S\)\\to\\mathcal\{G\}\(A\)\.

The anchoring component of classical RL is theBellman equation, which provides a recursive decomposition of the value function and enables agents to learn through successive approximation\. Central to this formulation is thevalue functionassociated with a policyπ\\pi, defined as the measurable functionVπ:S→ℝV^\{\\pi\}:S\\to\\mathbb\{R\}given by

Vπ​\(s\)=𝔼π​\[∑t=0∞γt​Rt\+1\|s0=s\],V^\{\\pi\}\(s\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{t\+1\}\\,\\middle\|\\,s\_\{0\}=s\\right\],\(1\)which represents the expected discounted return from statessunderπ\\pi\.

Building on this foundation, temporal\-difference \(TD\) learning introduces a paradigm shift by allowing agents to learn directly from raw experience in an online fashion, without requiring an explicit model of the environment dynamics\[[38](https://arxiv.org/html/2609.27745#bib.bib3)\]\. The core mechanism underlying TD learning is a bootstrapping update based on the TD error,

δt=Rt\+1\+γ​V​\(st\+1\)−V⁡\(st\),\\delta\_\{t\}=R\_\{t\+1\}\+\\gamma V\(s\_\{t\+1\}\)\-V\(s\_\{t\}\),\(2\)which quantifies the discrepancy between the current value estimate and a refined estimate incorporating newly observed rewards\.

From a categorical perspective, this learning process may be interpreted as an iterative search for a fixed point of the Bellman optimality operator, arising from the Kleisli structure induced by the Giry monad\. While theoretically robust, these classical approaches typically rely on tabular representations or simple function approximators, which are insufficient for capturing the complex, non\-linear structure present in high\-dimensional observation spaces\. This limitation motivates the use of more expressive models, such as deep neural networks, which can approximate rich value functions in such settings\.

Modern reinforcement learning systems combine these ideas with deep neural networks, but often require large quantities of data and struggle with generalisation and extrapolation\[[20](https://arxiv.org/html/2609.27745#bib.bib37),[35](https://arxiv.org/html/2609.27745#bib.bib14),[14](https://arxiv.org/html/2609.27745#bib.bib36)\]\. Recent work has therefore focused on incorporating structural inductive biases into RL systems, particularly those arising from symmetry and geometric structure\[[6](https://arxiv.org/html/2609.27745#bib.bib11),[44](https://arxiv.org/html/2609.27745#bib.bib19)\]\.

### 2\.2Symmetry and Equivariance

Symmetry can represent transformations capable of preserving structural properties, and thus are pivotal for expressing the invariances of a system\. In machine learning, symmetry appears as the equivariance of a model with respect to a group action\.

###### Definition 2\.3\(GG\-Equivariant Representation\)\.

LetGGbe a group acting on the state spaceSSvia⋅S\\cdot\_\{S\}and on a latent spaceZZvia⋅Z\\cdot\_\{Z\}\. A representation mappingf:S→Zf:S\\to ZisGG\-equivariant if for allg∈Gg\\in Gands∈Ss\\in S:

f⁡\(g⋅Ss\)=g⋅Zf⁡\(s\)f\(g\\cdot\_\{S\}s\)=g\\cdot\_\{Z\}f\(s\)

By restricting the hypothesis space to morphisms consistent with these structural constraints, agents ensure that the learned representationZZrespects the natural real\-world symmetries, improving generalisation\[[12](https://arxiv.org/html/2609.27745#bib.bib24),[6](https://arxiv.org/html/2609.27745#bib.bib11)\]\. Exploiting symmetry significantly reduces an agent’s hypothesis space, and by identifying key transformations that preserve the transition structure, the agent can effectively quotient the state spaceSSby the group action ofGG, focusing its learning capacity on the invariant features of the dynamics\. This mathematical advantage is why disentanglement is so vital for effectively separating the states by symmetry and improving sample efficiency: it allows an agent to generalise a single observation across an entire symmetry orbit, ensuring that the internal model remains consistent even as the specific external state changes\.

### 2\.3Disentanglement through Symmetry\-Based Decomposition

While global symmetries provide potential for better representations, disentanglement is necessary to effectively partition an environment by its symmetries\. By aligning internal representations with independent physical factors of the environment, an agent can capture specific environmental details such as spatial coordinates, object geometries, or dynamic velocities without the risk of crosstalk\[[4](https://arxiv.org/html/2609.27745#bib.bib41)\]\.

Higgins et al\. formalise this process as the structural alignment between latent variables and the underlying symmetry transformations of the data\-generating process\[[22](https://arxiv.org/html/2609.27745#bib.bib4)\]\. Categorically, this represents a shift from a monolithic state objectSSto a structured latent objectZZthat admits aproduct decomposition:

Z≅∏i=1nZi,G≅∏i=1nGiZ\\cong\\prod\_\{i=1\}^\{n\}Z\_\{i\},\\quad G\\cong\\prod\_\{i=1\}^\{n\}G\_\{i\}\(3\)where each subgroupGiG\_\{i\}acts non\-trivially only on its corresponding factorZiZ\_\{i\}\.

This reconstruction comes with key mathematical benefits\. First, it enables the factorisation of the transition morphismα:Z→𝒢⁡\(A×Z×ℝ\)\\alpha:Z\\to\\mathcal\{G\}\(A\\times Z\\times\\mathbb\{R\}\)so if a specific actiona∈Aa\\in Aonly affects the coordinateZp​o​sZ\_\{pos\}, the agent can model this transformation as the identity on all other factorsZj≠p​o​sZ\_\{j\\neq pos\}\. Second, such disentangled representations improve generalisation by localising the effect of variations in individual factors, thereby reducing the impact of spurious correlations in out\-of\-distribution settings\[[35](https://arxiv.org/html/2609.27745#bib.bib14)\]\. This structured separation allows the agent to propagate information selectively across states that share relevant underlying factors, leading to more stable and reliable value estimation\.

### 2\.4Global Symmetry via Group Actions

Groups provide one canonical formalism for describing global symmetries, as groups can act on entire state spaces to represent transformations that preserve underlying structural properties of an environment\. A groupGGis a set with a binary operation satisfying associativity, identity, and invertibility\[[30](https://arxiv.org/html/2609.27745#bib.bib49)\]\.

###### Definition 2\.4\(Group Action and Equivariance\)\.

A left group action ofGGon a setXXis a map⋅:G×X→X\\cdot:G\\times X\\to Xsatisfyinge⋅x=xe\\cdot x=xandg⋅\(h⋅x\)=\(g​h\)⋅xg\\cdot\(h\\cdot x\)=\(gh\)\\cdot x\. A morphismf:X→Yf:X\\to YbetweenGG\-sets isGG\-equivariant if it commutes with the group action:

f⁡\(g⋅x\)=g⋅f⁡\(x\)∀g∈G,x∈X\.f\(g\\cdot x\)=g\\cdot f\(x\)\\quad\\forall g\\in G,x\\in X\.

Neural architectures constrained by these global symmetries leverage this equivariance to achieve superior sample efficiency by sharing parameters across the group orbit𝒪x=\{g⋅x∣g∈G\}\\mathcal\{O\}\_\{x\}=\\\{g\\cdot x\\mid g\\in G\\\}\[[12](https://arxiv.org/html/2609.27745#bib.bib24)\]\. However, global equivariance implies that the transition dynamics are uniform across the entire state spaceSS, which is unrepresentative of realistic partial local symmetries\. For example, rotational symmetries that exist an open, unobstructed states∈So​p​e​ns\\in S\_\{open\}is destroyed if the agent enters a coordinatesswith environmental boundaries\. This transition from global to state\-dependent symmetry necessitates a framework where the set of valid transformations𝒢⁡\(s,s′\)\\mathcal\{G\}\(s,s^\{\\prime\}\)depends on the specific states being related, rather than a single, all\-encompassing groupGG, such as withgroupoids\.

### 2\.5Local Symmetries via Groupoid Actions

A groupoid generalises the notion of a group by allowing symmetries to exist as morphisms between distinct objects rather than a single identity\-centered set\. This allows us to model transformations that are only composable when their context matches under some conditions, capturing partial symmetries\.

###### Definition 2\.5\(Groupoid\)\.

A groupoid𝒢\\mathcal\{G\}is a small category in which every morphism is an isomorphism\. It consists of a set of objects𝒢0\\mathcal\{G\}\_\{0\}and a set of arrows𝒢1\\mathcal\{G\}\_\{1\}, equipped with source and target mapss,t:𝒢1→𝒢0s,t:\\mathcal\{G\}\_\{1\}\\to\\mathcal\{G\}\_\{0\}\. For any two arrowsf,g∈𝒢1f,g\\in\\mathcal\{G\}\_\{1\}, the compositiong∘fg\\circ fis defined if and only ifs⁡\(g\)=t⁡\(f\)s\(g\)=t\(f\)\.

In an RL context,𝒢0\\mathcal\{G\}\_\{0\}may represent the set of possible environmental states, while𝒢1\\mathcal\{G\}\_\{1\}represents the reversible transformations between them\[[48](https://arxiv.org/html/2609.27745#bib.bib2)\]\. If two independent subsystems have state spacesS1S\_\{1\}andS2S\_\{2\}, their joint environment is naturally represented by the product objectS1×S2S\_\{1\}\\times S\_\{2\}, with transition dynamics constructed functorially from the component kernels\[[16](https://arxiv.org/html/2609.27745#bib.bib46)\]\. Groupoids therefore provide a pathway for decomposing complex environments into simpler components using category theory\.

## 3Groupoid Reinforcement Learning

This section will utilise the mathematical theory to achieve a computational implementation, starting with an overview of the approach and an outline of the algorithm, and will be examining two canonicalisation method for two types of RL solutions\.

### 3\.1Groupoid Solver Overview

The interaction begins with a generative cycle: the agent receives an observationoto\_\{t\}and a rewardrtr\_\{t\}from the environment\. Rather than performing a standard Bayesian update on a high\-dimensional belief manifold, the agent invokes acanonicalisation morphism\. This morphism maps the raw, perceived state into its unique orbit representativest−∈S−s^\{\-\}\_\{t\}\\in S^\{\-\}, while simultaneously recording thetransporterelementτ∈G\\tau\\in Gthat tracks the spatial orientation of the original state\. In classical reinforcement learning, the agent maintains aQ\-table, a functionQ:S×A→ℝQ:S\\times A\\to\\mathbb\{R\}that stores the expected return of taking actionaain statess\. In our setting, this table is defined over the canonical space,Q:S−×A−→ℝQ:S^\{\-\}\\times A^\{\-\}\\to\\mathbb\{R\}, so that each entry corresponds not to a single raw state, but to an entire equivalence class of symmetric configurations\.

This architecture therefore mitigates the curse of dimensionality through symmetry\-induced parameter sharing\. Updates computed at a canonical representatives−s^\{\-\}are propagated across all states isomorphic tos−s^\{\-\}, allowing each interaction to inform an entire equivalence class\. When learning in a compressed canonical space, actions in the original state space are obtained via transport throughτs\\tau\_\{s\}, and by leveraging structured correspondences, each transition contributes to a wider update of the value function, improving sample efficiency and supporting learning in sparse\-reward environments\. Under standard conditions, such update rules inherit the convergence guarantees of Q\-learning\[[45](https://arxiv.org/html/2609.27745#bib.bib50)\]\.

### 3\.2Algorithm

Here we outline the groupoid\-aware Q\-learning algorithm\. We decouple the symmetry logic into a separate canonicalisation interface to allow for modular implementation and testing through one interface\.

Algorithm 1Groupoid Q\-Learning Agent with canonicalisation and local symmetry propagationClass*GroupoidQLearner\(α,γ,ϵ,𝒞\\alpha,\\gamma,\\epsilon,\\mathcal\{C\}\)*

//

𝒞\\mathcal\{C\}is an external Canonicalisation Method

Q←Q\\leftarrowempty object\-level Q\-table

Function*choose\_action\(ss\)*

\(s−,τs\)←𝒞\.get\_canonical​\(s\)\(s^\{\-\},\\tau\_\{s\}\)\\leftarrow\\mathcal\{C\}\.\\text\{get\\\_canonical\}\(s\)
a−∼ϵ​\-greedy​\(Q⁡\[s−\]\)a^\{\-\}\\sim\\epsilon\\text\{\-greedy\}\(Q\[s^\{\-\}\]\)
a←τs−1⋅a−a\\leftarrow\\tau\_\{s\}^\{\-1\}\\cdot a^\{\-\}
return

aa
Function*update\(s,a,r,s′s,a,r,s^\{\\prime\}\)*

\(s−,τs\)←𝒞\.get\_canonical​\(s\)\(s^\{\-\},\\tau\_\{s\}\)\\leftarrow\\mathcal\{C\}\.\\text\{get\\\_canonical\}\(s\)
\(s′⁣−,τs′\)←𝒞\.get\_canonical​\(s′\)\(s^\{\\prime\-\},\\tau\_\{s^\{\\prime\}\}\)\\leftarrow\\mathcal\{C\}\.\\text\{get\\\_canonical\}\(s^\{\\prime\}\)
a−←τs⋅aa^\{\-\}\\leftarrow\\tau\_\{s\}\\cdot a
//Update canonical Q\-value

Q⁡\[s−,a−\]←Q⁡\[s−,a−\]\+α⁡\(r\+γ​maxa′⁡Q⁡\[s′⁣−,a′\]−Q⁡\[s−,a−\]\)Q\[s^\{\-\},a^\{\-\}\]\\leftarrow Q\[s^\{\-\},a^\{\-\}\]\+\\alpha\\big\(r\+\\gamma\\max\_\{a^\{\\prime\}\}Q\[s^\{\\prime\-\},a^\{\\prime\}\]\-Q\[s^\{\-\},a^\{\-\}\]\\big\)
//Symmetry Propagation via Groupoid Morphisms

foreach*morphismϕ:s−→t\\phi:s^\{\-\}\\to tin𝒞\.orbit​\(s−\)\\mathcal\{C\}\.\\text\{orbit\}\(s^\{\-\}\)*do

Q⁡\[t,ϕ⁡\(a−\)\]←Q⁡\[t,ϕ⁡\(a−\)\]\+α⁡\(r\+γ​maxa′⁡Q⁡\[ϕ⁡\(s′⁣−,a′\)\]−Q⁡\[t,ϕ⁡\(a−\)\]\)Q\[t,\\phi\(a^\{\-\}\)\]\\leftarrow Q\[t,\\phi\(a^\{\-\}\)\]\+\\alpha\\big\(r\+\\gamma\\max\_\{a^\{\\prime\}\}Q\[\\phi\(s^\{\\prime\-\},a^\{\\prime\}\)\]\-Q\[t,\\phi\(a^\{\-\}\)\]\\big\)
Class*Canonicaliser*

Function*get\_canonical\(ss\)*

//Return representatives−s^\{\-\}and transporterτs:s−→s\\tau\_\{s\}:s^\{\-\}\\to s

return

\(s−,τs\)\(s^\{\-\},\\tau\_\{s\}\)
Function*orbit\(s−s^\{\-\}\)*

returnset of morphisms

\{ϕi\}\\\{\\phi\_\{i\}\\\}from

s−s^\{\-\}to isomorphic states

### 3\.3Functorial State Reductions via Canonicalisation

An effective Groupoid Q\-Learner needs an effective methodology to partition a spaceSSto a minimal set of orbits and their representatives\. We define an abstract canonicalisation morphism𝒞:S→S−\\mathcal\{C\}:S\\to S^\{\-\}, and implement two different partitioning methods using structuralGG\-equivalence and dynamical bisimulation\.

These two canonicalisation methods offer complementary ways to exploit structure in reinforcement learning and thus represent two different scenarios for the application of category theory in RL\. Predefined symmetries inject prior knowledge to improve efficiency and are useful in controlled environments, while discovered symmetries extract latent regularities from data, enabling generalisation when the symmetry group is unknown or approximate\.

#### 3\.3\.1Structural Canonicalisation:GG\-Orbit Quotients

In the structural approach, we assume a groupoid𝒢⇉S\\mathcal\{G\}\\rightrightarrows Sencoding local symmetries between states: two statess,s′s,s^\{\\prime\}are considered isomorphic if there exists a morphismγ:s→s′\\gamma:s\\to s^\{\\prime\}in𝒢\\mathcal\{G\}\.

###### Definition 3\.1\(Structural Isomorphism via Groupoids\)\.

For𝒢⇉S\\mathcal\{G\}\\rightrightarrows Sa \(small\) groupoid with object setSS, the canonicalisation𝒞\\mathcal\{C\}is a choice of representatives for isomorphism classes, a projection onto the orbit spaceπ0​\(𝒢\)\\pi\_\{0\}\(\\mathcal\{G\}\)\. For any states∈Ss\\in S, the canonicaliser returns a pair\(s−,τs\)\(s^\{\-\},\\tau\_\{s\}\)such that:

1. 1\.s−s^\{\-\}is the chosen representative of the connected component \(orbit\): 𝒪s=\{s′∈S∣∃γ:s−→s′in𝒢\}\\mathcal\{O\}\_\{s\}=\\\{\\,s^\{\\prime\}\\in S\\mid\\exists\\gamma:s^\{\-\}\\to s^\{\\prime\}\\text\{ in \}\\mathcal\{G\}\\,\\\}
2. 2\.τs:s−→s\\tau\_\{s\}:s^\{\-\}\\to sis atransportermorphism in𝒢\\mathcal\{G\}\.

This formulation replaces a global group action with a local symmetry structure encoded by𝒢\\mathcal\{G\}, allowing different states to admit different symmetry groups that are predefined\. The burden of specifying symmetry is thus shifted to defining the groupoid structure itself\.

This method effectively puts the burden of symmetry definition on the system designer, leveraging a priori geometric knowledge to get best results but requiring significant fine\-tuning by a designer ahead of time\. For example, in an autonomous urban navigation task, objects are intersectionss∈Ss\\in S, and morphisms in𝒢\\mathcal\{G\}correspond to local rigid motions \(translations and rotations\), mapping valid intersections to each other\. If two intersectionsssands′s^\{\\prime\}are connected by a morphism in𝒢\\mathcal\{G\}, they lie in the same connected component and hence share the same canonical representatives−s^\{\-\}, so updating a value function ats−s^\{\-\}propagates information across all symmetrical states, improving sample efficiency while respecting only those symmetries that are locally valid\.

#### 3\.3\.2Dynamical Canonicalisation: Coalgebraic Bisimulation

Structural symmetry can also be found procedurally, utilising environmental structure to derive canonicalisation from the transition dynamics, using the reward values and state properties to define symmetry\. This is achieved by viewing the POMDP as a coalgebraα:S→𝒢​\(O×R×S\)A\\alpha:S\\to\\mathcal\{G\}\(O\\times R\\times S\)^\{A\}for the Giry monad𝒢\\mathcal\{G\}\.

###### Definition 3\.2\(δ\\delta\-Granular Bisimulation\)\.

Two statess,s′s,s^\{\\prime\}are dynamicallyδ\\delta\-equivalent if their transition measures are close under a metricdd\. Formally, we defines≈δs′s\\approx\_\{\\delta\}s^\{\\prime\}if for all actionsa∈Aa\\in A:

d𝒢​\(α⁡\(s,a\),α⁡\(s′,a\)\)<δd\_\{\\mathcal\{G\}\}\\left\(\\alpha\(s,a\),\\alpha\(s^\{\\prime\},a\)\\right\)<\\deltawhereδ∈\[0,1\]\\delta\\in\[0,1\]is a granularity parameter governing the resolution of the state\-reward manifold\.

Under this regime, the canonicalisation method𝒞\\mathcal\{C\}will define states with similar reward behaviour as symmetrical, even if they lack a clear geometric relationship\. The granularity of this approach can be determined by alteringδ\\delta: asδ→0\\delta\\to 0, we recover exact bisimulation; and asδ\\deltaincreases, the agent performs a lossy compression of the environment\.

Using the aforementioned urban navigation example, the agent will this time behind with no explicit information or inherent spatial awareness, and instead derives these symmetries from the ground up by observing its environment\. For example, it may realise that the transition probabilities and reward signals of various intersections are identical, and thus they are symmetrical\.

While both methods may ultimately converge on the same efficient policy by collapsing an environment into its essential topological representatives, the structural approach bypasses the discovery phase through human\-led geometric priors, whereas the automatic approach provides a robust, data\-driven fail\-safe for environments where the symmetry may be irregular or hidden from the designer’s view\.

### 3\.4Symmetry\-Aware Temporal Difference Learning

In our framework, the Q\-function extends beyond a tabular representation by explicitly incorporating the symmetry structure of the state space\. Rather than updating a single state\-action pair, each observed transition\(s,a,r,s′\)\(s,a,r,s^\{\\prime\}\)induces a symmetry\-aware temporal\-difference update that propagates across all states related tossunder the underlying symmetry structure\. By sharing information across these equivalent configurations, the learning process reduces redundancy and improves sample efficiency\.

This symmetry\-aware update rule is as follows: let𝒞:S→S^\\mathcal\{C\}:S\\to\\hat\{S\}be a canonicalisation mapping that assigns each statessa representatives−∈S^s^\{\-\}\\in\\hat\{S\}, together with an isomorphism \(transporter\)τs:s−→s\\tau\_\{s\}:s^\{\-\}\\to s\. The canonical Q\-valueQ⁡\(s−,a−\)Q\(s^\{\-\},a^\{\-\}\)is updated using the standard Bellman residual, and this update is then propagated to any statettisomorphic tos−s^\{\-\}via a groupoid morphismϕ:s−→t\\phi:s^\{\-\}\\to t:

∀ϕ∈Mor​\(s−,t\),Q⁡\(t,ϕ⁡\(a−\)\)←Q⁡\(t,ϕ⁡\(a−\)\)\+α⋅δ\\forall\\phi\\in\\text\{Mor\}\(s^\{\-\},t\),\\quad Q\(t,\\phi\(a^\{\-\}\)\)\\leftarrow Q\(t,\\phi\(a^\{\-\}\)\)\+\\alpha\\cdot\\delta\(4\)
where the TD\-errorδ\\deltais pulled back from the canonical transition:

δ=r\+γ​maxa′⁡Q⁡\(s′⁣−,a′\)−Q⁡\(s−,a−\)\.\\delta=r\+\\gamma\\max\_\{a^\{\\prime\}\}Q\(s^\{\\prime\-\},a^\{\\prime\}\)\-Q\(s^\{\-\},a^\{\-\}\)\.\(5\)
The mechanism ensures that the policyπ\\piisGG\-equivariant and therefore the agent only ever learns in the simplified canonical spaceS−S^\{\-\}, and only acts in the raw spaceSSby pulling back the optimal action through the inverse transporterτs−1\\tau\_\{s\}^\{\-1\}:

s\{\\lx@inpgf@ignorespaces s\}a∈A⁡\(s\)\{\\lx@inpgf@ignorespaces a\\in A\(s\)\}s−\{\\lx@inpgf@ignorespaces s^\{\-\}\}a−∈A⁡\(s−\)\{\\lx@inpgf@ignorespaces a^\{\-\}\\in A\(s^\{\-\}\)\}τs−1\\scriptstyle\{\\lx@inpgf@ignorespaces\\tau\_\{s\}^\{\-1\}\}Q\\scriptstyle\{\\lx@inpgf@ignorespaces Q\}τs\\scriptstyle\{\\lx@inpgf@ignorespaces\\tau\_\{s\}\}

Our approach propagates the rewardrracross all states symmetrical states in its orbit, and thus a single interaction informs the agent over the entire equivalence class\. This resulting groupoid\-based Q\-learning framework leverages local symmetries to augment efficiency by adapting to state\-dependant symmetries and allowing the model to generalise where appropriate without oversimplifying the dynamics\. Importantly, the separation between representation via the groupoid𝒢\\mathcal\{G\}and canonicalisation𝒞\\mathcal\{C\}and learning dynamics enables controlled empirical analysis of how symmetry structure influences performance\.

## 4Empirical Evaluation

This section will first outline the environments, parameters and baseline used to evaluate performance, and then discuss the results of the evaluation and the implications\.

### 4\.1POMDPy Environments

POMDPy is a Python framework for building and evaluating reinforcement learning algorithms in POMDP environments\. It offers high\-level abstractions for defining the essential components of a POMDP tuple and provides comprehensive implementable code that allows for a seamless transition from category theory to algorithmic implementation\. Our evaluation usesh2r/POMDP\-Py, a fork from Brown University’s Human to Robot \(h2r\) Laboratory, whose modular design separates the environment’s generative model from belief\-tracking and planning, enabling direct comparison between our categorical formulation and the repository’s native high\-performance solvers in two benchmark environments:RockSampleandTag\.

### 4\.2RockSample

RockSampleis a canonical POMDPy benchmark designed to evaluate exploration under uncertainty, comprising a grid world containing several rocks, each with an unknown binary quality \(good or bad\), hidden from the agent\. The agent may move in the cardinal directions or perform one of two interactions: it cansensea rock, receiving a noisy observation whose accuracy increases with proximity, with probability

P⁡\(correct∣d\)=1\+2−d/d02,P\(\\text\{correct\}\\mid d\)=\\frac\{1\+2^\{\-d/d\_\{0\}\}\}\{2\},whereddis the distance between the agent and the rock, andd0d\_\{0\}is an environment constant \(typicallyd0=20d\_\{0\}=20\); alternatively, it cansamplea rock at its current location, receiving a positive reward if the rock is good and a negative reward otherwise\.

The primary challenge arises from the trade\-off between exploration and exploitation\. Moving closer to a rock before sensing reduces observational uncertainty but incurs a time cost, while sampling yields immediate reward but risks selecting a bad rock\. The environment thus tests spatial reasoning and belief\-state planning in partially observable environments\.

Exit AreaAR1R\_\{1\}R2R\_\{2\}R3R\_\{3\}R4R\_\{4\}Sense\(R2R\_\{2\}\)Move EastAAgentRiR\_\{i\}Rock \(unknown quality\)Exit areaSense actionMovement action

Figure 1:Small5×55\\times 5example
RockSamplegeneration\.

### 4\.3Tag

TheTagenvironment models a classic pursuit\-evasion problem under partial observability, in which an agent must locate and capture a moving target within a discrete grid world\. The agent’s position is fully observable, while the target’s position is hidden and must be inferred from limited observations that provide coarse information about its location\. At each timestep, the agent may choose to remain stationary, move in one of the cardinal directions, or execute atagaction\. A successful tag yields a positive reward if the agent occupies the same cell as the target, and a penalty otherwise, discouraging premature or uninformed tagging\.

The target evolves independently according to a stochastic transition model, and the agent receives only partial information about its position\. As a result,Tagemphasises the challenge of tracking a hidden, dynamically moving object through sequential inference\. It serves as a benchmark for belief\-state estimation and pursuit under uncertainty, requiring the agent to integrate information over time while balancing exploration with timely action to achieve successful capture\.

ATP=0\.1P=0\.1P=0\.1P=0\.1P=0\.8P=0\.8Move NorthCertain Sensing RangeAAgentTTargetStochastic transition \(withPP\)Agent actionCertain Sensing range

Figure 2:Small5×45\\times 4exampleTag
generation\.

### 4\.4Benchmarks and configurations

We benchmark our symmetry\-aware representations against Partially Observable Monte Carlo Planning \(POMCP\), which uses Monte Carlo Tree Search \(MCTS\) to approximate optimal policies without explicit belief\-space representations\. As a model\-free, simulation\-based planner, POMCP operates over histories and uses particle filtering to implicitly represent beliefs, making it a strong baseline for high\-dimensional POMDPs\. Its asymptotic optimality and lack of structural assumptions contrasts to our approach, and our comparisons examine if symmetry yields gains over general\-purpose models\.

We conducted a series of tests within a20×2020\\times 20RockSampleconfiguration and a15×1515\\times 15Tagto observe multi\-step information gathering in large state spaces, and how agents manage belief manifolds over large complex environments\. Comprehensive parameters can be found in Table[1](https://arxiv.org/html/2609.27745#S4.T1)\.

Table 1:Comprehensive Environmental Parameters and Agent Configuration Matrix
### 4\.5Results

#### 4\.5\.1Knowledge Propagation in High\-Entropy

![[Uncaptioned image]](https://arxiv.org/html/2609.27745v1/Images/rocksample_entropy_trimmed.png)

Figure 3:Exploration Entropy in20×2020\\times 20RockSample, with high\-intensity regions showing verified rock locations, and blue regions showing areas benefitting from reflected learning\.
Testing in the20×2020\\times 20RockSampleenvironment highlights how interactions between the sensor efficiency constantd0=20d\_\{0\}=20and the grid scale creates a significant information bottleneck\. Unlike the baseline, which must physically visit each location to reduce uncertainty, the structured approach uses the transporterτs\\tau\_\{s\}to map a single observation at\(x,y\)\(x,y\)to its symmetric counterparts\. As shown in Figure[3](https://arxiv.org/html/2609.27745#S4.F3), this produces clusters of high knowledge density in the blue, unvisited regions of the grid as shared learning from results observed elsewhere in the class are propagated across entire equivalence classes\. Conversely, the agent leverages local observations to reduce uncertainty across the state space, partially overcoming thed0d\_\{0\}sensing limitation and enabling it to inference of distant rock values through structural similarity\.

#### 4\.5\.2Learning Dynamics, Convergence Rates and Belief Diffusion

A comparative analysis of the learning curves highlights a clear hierarchy of efficiency across environments of increasing complexity\. Figure[4](https://arxiv.org/html/2609.27745#S4.F4)compares cumulative reward over time for both the20×2020\\times 20RockSampleand15×1515\\times 15Tagdomains, showing that the groupoid\-based approaches consistently outperform the POMCP baseline by mitigating the sparse\-reward challenges inherent to high\-dimensional POMDPs, and illustrate the relative differences between our two approaches\.

![Refer to caption](https://arxiv.org/html/2609.27745v1/Images/rock_learning.png)\(a\)Comparison of model reward forRockSample\.
![Refer to caption](https://arxiv.org/html/2609.27745v1/Images/tag_learning.png)\(b\)Comparison of model reward forTag\.

Figure 4:These subfigures show the relative performance of three different models: the structural and automatic groupoid learners, as well as the baseline POMCP method, across two key POMDP systems\.The data fromRockSample\(Figure[4\(a\)](https://arxiv.org/html/2609.27745#S4.F4.sf1)\) shows the structural groupoid learner rapidly achieving a high success rate, stabilizing near a reward of2525within approximately500500episodes, while the automatic approach stabilises slightly lower, around2323\. The steep initial learning curve reflects the effect of the DihedralD4D\_\{4\}groupoid action, which propagates eachQQ\-value update across all symmetric states in an equivalence class simultaneously\. This group consists of the eight symmetries of a square, including four rotations and four reflections and using it captures all transformations under which the environment dynamics and rewards remain invariant\. We thus use this to predefine symmetries within the environment for each rock, illustrating how the structured approach allows for efficient symmetries to be encoded provided that the designer has sufficient knowledge of the environment\. This symmetry\-induced sharing enables generalisation from sparse local interactions, significantly accelerating early\-stage learning\.

Compared to this, the automatic bisimulation method exhibits an initial discovery lag as it constructs its own understanding of the environment usingδ\\delta\-granular partitions before it is able to experience shared learning and converge close to the structural approach, demonstrating that symmetry can be effectively inferred rather than explicitly specified\. This highlights the robustness of the coalgebraic abstraction even when structural knowledge is not provided a priori\.

Conversely, the learning curves for the15×1515\\times 15Tagenvironment \(Figure[4\(b\)](https://arxiv.org/html/2609.27745#S4.F4.sf2)\) are inherently more erratic\.Tagpresents a higher\-entropy setting, with\|S\|\>50,000\|S\|\>50\{,\}000joint states and a dynamically evolving target, leading to significant variance in observed rewards\. The sparsity and binary nature of the reward signal \(successful capture versus failure\) introduce sharp fluctuations, as missed tagging opportunities can rapidly diffuse the agent’s belief over the target’s location\.

Overall, both methods consistently outperform the baseline and maintain higher rewards throughout training\. Furthermore, the500500episode limit is insufficient for traditional methods such as the baseline POMCP approach to leave its initial exploration phase in juxtaposition to our approach which is able to accelerate learning and achieve results faster\. This demonstrates that the advantages of canonicalisation persist even in highly stochastic environments, where uncertainty is continually reintroduced and must be managed over time\.

## 5Conclusions and Future Work

The empirical evaluations conducted in the20×2020\\times 20RockSampleand15×1515\\times 15Tagdomains demonstrate that the Groupoid Q\-Learner effectively mitigates the curse of dimensionality and the challenges of partial observability by exploiting environmental symmetries\. By collapsing a vast state\-action manifold into a minimal canonical representation, the learner achieves a substantial reduction in sample complexity, reaching reward stability up to 45% faster than the standard POMCP baseline\. Our analysis of knowledge propagation shows that structural canonicalisation enables global value updates from local observations, allowing the agent to generalise across isomorphic states and reduce uncertainty even in high\-entropy regimes where sensing is limited\. In effect, the groupoid formulation treats symmetric states as a single computational object, ensuring that experience gained in one region of the environment is immediately transferable across structurally equivalent regions\. Furthermore, the automated bisimulation procedure demonstrates that such structure need not be hand\-engineered: the agent is able to discover and exploit latent symmetries directly from interaction, highlighting the practical utility of the approach in environments where prior knowledge is unavailable\.

Beyond empirical performance, this work underscores a broader methodological point: category theory is not merely a descriptive language for organising abstractions, but a constructive tool for designing learning algorithms\. By explicitly encoding symmetry through groupoids and canonicalisation, we obtain a principled mechanism for resolving core reinforcement learning challenges including parameter sharing, credit assignment, and generalisation\. The resulting framework demonstrates how categorical structure can translate directly into measurable gains in efficiency and robustness, particularly in settings where traditional methods struggle with combinatorial or high\-dimensional complexity\.

Future research directions include extending the groupoid framework to continuous state\-action spaces, where discrete symmetries give way to smooth transformations such as Lie group actions\. This will require the development of equivariant function approximators capable of learning transporter morphisms over continuous domains\. Another promising direction is the application of groupoid structure to hierarchical reinforcement learning: the inherently compositional nature of groupoids suggests a natural bridge to higher\-dimensional categorical frameworks, such asnn\-categories, where symmetries can be organised across multiple levels of abstraction\. Such a perspective may enable agents to reason not only about state\-level symmetries, but also about reusable substructures and policies at different scales\.Additionally, adaptive control of the partition granularityδ\\deltaoffers a pathway to balancing abstraction with precision, while extensions to multi\-agent settings may leverage shared structural morphisms to facilitate coordination under partial observability\.

Ultimately, the success of the Groupoid Q\-Learner across these domains demonstrates the power of viewing environments through the lens of symmetry and structure\. By grounding learning dynamics in categorical principles, we obtain a framework that is both mathematically rigorous and practically effective, offering a compelling direction for scaling reinforcement learning to increasingly complex and uncertain domains\.

## References

- \[1\]J\. C\. Baez and J\. Dolan\(2001\)From finite sets to feynman diagrams\.Mathematics Unlimited—2001 and Beyond\.
- \[2\]E\. J\. Bekkers\(2019\)Roto\-translation covariant convolutional networks for medical image analysis\.InMedical Image Computing and Computer Assisted Intervention – MICCAI 2019,Lecture Notes in Computer Science, Vol\.11767,pp\. 440–448\.
- \[3\]E\. A\. Ben Opperman and E\. Mondragón\(2026\)Groupoid\-based internal state representations for reinforcement learning with local symmetries\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2026\),pp\. 1–9\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p3.1)\.
- \[4\]Y\. Bengio, A\. Courville, and P\. Vincent\(2013\)Representation learning: a review and new perspectives\.IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\),pp\. 1798–1828\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27745#S2.SS3.p1.1)\.
- \[5\]M\. Botvinick, S\. Ritter, J\. X\. Wang, Z\. Kurth\-Nelson, C\. Blundell, and D\. Hassabis\(2019\)Reinforcement learning, fast and slow\.Trends in Cognitive Sciences23\(5\),pp\. 408–422\.
- \[6\]M\. M\. Bronstein, J\. Bruna, T\. S\. Cohen, and P\. Veličković\(2021\)Geometric deep learning: grids, groups, graphs, geodesics, and gauges\.MIT Press,Cambridge, MA\.Cited by:[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p7.1),[§2\.2](https://arxiv.org/html/2609.27745#S2.SS2.p2.1)\.
- \[7\]R\. Brown\(2006\)Topology and groupoids\.BookSurge\.
- \[8\]M\. Campbell, A\. J\. Hoane, and F\. Hsu\(2002\)Deep Blue\.Artificial Intelligence134\(1–2\),pp\. 57–83\.
- \[9\]H\. Caselles\-Dupré, A\. Garcia\-Ortiz, and D\. Filliat\(2019\)Symmetry\-based disentangled representation learning requires interaction with environments\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 4601–4612\.
- \[10\]H\. Caselles\-Dupré, M\. Rolinek, J\. Von Kries, D\. Bouchacourt, M\. Kumar, and P\. Kohli\(2020\)Linear disentangled representations and unsupervised action estimation\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 13000–13011\.
- \[11\]T\. S\. Cohen, M\. Geiger, J\. Köhler, and M\. Welling\(2019\)Gauge equivariant convolutional networks and the icosahedral CNN\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 1321–1330\.
- \[12\]T\. Cohen and M\. Welling\(2016\)Group equivariant convolutional networks\.InProceedings of the 33rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.48,pp\. 2990–2999\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.27745#S2.SS2.p2.1),[§2\.4](https://arxiv.org/html/2609.27745#S2.SS4.p2.1)\.
- \[13\]A\. Dean, E\. Alonso, and E\. Mondragón\(2025\)Algebras of actions in an agent’s representations of the world\.Artificial Intelligence348,pp\. 104403\.External Links:[Document](https://dx.doi.org/10.1016/j.artint.2025.104403)
- \[14\]G\. Dulac\-Arnold, N\. Levine, D\. J\. Mankowitz, J\. Li, C\. Paduraru, S\. Gowal, and T\. Hester\(2021\)Challenges of real\-world reinforcement learning\.Machine Learning110\(9\),pp\. 2429–2472\.Cited by:[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p7.1)\.
- \[15\]R\. Evans and J\. Gao\(2016\)DeepMind AI reduces Google data centre cooling bill by 40%\.Note:DeepMind BlogExternal Links:[Link](https://deepmind.google/discover/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-40/)
- \[16\]B\. Fong and D\. I\. Spivak\(2019\)Seven sketches in compositionality: an invitation to applied category theory\.MIT Press\.Note:Available at arXiv:1803\.05316Cited by:[§2\.5](https://arxiv.org/html/2609.27745#S2.SS5.p2.1)\.
- \[17\]M\. Giry\(1982\)A categorical approach to probability theory\.InCategorical Aspects of Topology and Analysis,Lecture Notes in Mathematics, Vol\.915,pp\. 68–85\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p2.1)\.
- \[18\]O\. Gottesman, F\. Johansson, J\. Meier, J\. Dent, D\. Lee, S\. Srinivasan, L\. Zhang, Y\. Ding, D\. Wihl, X\. Peng,et al\.\(2019\)Guidelines for reinforcement learning in healthcare\.Nature Medicine25\(1\),pp\. 16–18\.
- \[19\]S\. Gu, E\. Holly, T\. Lillicrap, and S\. Levine\(2017\)Deep reinforcement learning for robotic manipulation with asynchronous off\-policy updates\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 3389–3396\.
- \[20\]P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. Meger\(2018\)Deep reinforcement learning that matters\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32,pp\. 3207–3214\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p7.1)\.
- \[21\]M\. Hessel, J\. Modayil, H\. van Hasselt, T\. Schaul, G\. Ostrovski, W\. Dabney, D\. Horgan, B\. Piot, M\. Azar, and D\. Silver\(2018\)Rainbow: combining improvements in deep reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32,pp\. 3215–3222\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v32i1.11796)
- \[22\]I\. Higgins, L\. Matthey, A\. Pal, C\. Burgess, X\. Glorot, M\. Botvinick, S\. Mohamed, and A\. Lerchner\(2018\)Towards a definition of disentangled representations\.arXiv preprint arXiv:1812\.02230\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27745#S2.SS3.p2.1)\.
- \[23\]Y\. Hu, H\. Ma, B\. Dai, and N\. Li\(2024\)Efficient duple perturbation robustness in low\-rank Markov decision processes\.arXiv preprint arXiv:2404\.08089\.
- \[24\]B\. Jacobs\(2016\)Introduction to coalgebra: towards mathematics of states and observation\.Cambridge Tracts in Theoretical Computer Science,Cambridge University Press,Cambridge\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p2.1)\.
- \[25\]J\. Kober, J\. A\. Bagnell, and J\. Peters\(2013\)Reinforcement learning in robotics: a survey\.The International Journal of Robotics Research32\(11\),pp\. 1238–1274\.
- \[26\]M\. Komorowski, L\. A\. Celi, O\. Badawi, A\. C\. Gordon, and A\. A\. Faisal\(2018\)The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care\.Nature Medicine24\(11\),pp\. 1716–1720\.
- \[27\]B\. M\. Lake, T\. D\. Ullman, J\. B\. Tenenbaum, and S\. J\. Gershman\(2017\)Building machines that learn and think like people\.Behavioral and Brain Sciences40,pp\. e253\.
- \[28\]Y\. LeCun, L\. Bottou, Y\. Bengio, and P\. Haffner\(1998\)Gradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.
- \[29\]S\. Levine, C\. Finn, T\. Darrell, and P\. Abbeel\(2016\)End\-to\-end training of deep visuomotor policies\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 1334–1342\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p1.1)\.
- \[30\]S\. Mac Lane\(1998\)Categories for the working mathematician\.2 edition,Springer\.Cited by:[§2\.4](https://arxiv.org/html/2609.27745#S2.SS4.p1.1)\.
- \[31\]H\. Mao, M\. Alizadeh, I\. Menache, and S\. Kandula\(2016\)Resource management with deep reinforcement learning\.InProceedings of the 15th ACM Workshop on Hot Topics in Networks \(HotNets\),pp\. 50–56\.
- \[32\]E\. Noether\(1918\)Invariante Variationsprobleme\.Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathematisch\-Physikalische Klasse1918,pp\. 235–257\.Note:English translation: Invariant Variation ProblemsCited by:[§1](https://arxiv.org/html/2609.27745#S1.p3.1)\.
- \[33\]M\. Painter\(2022\)On the learning and structure of Symmetry Based Disentangled Representations\.Ph\.D\. Thesis,University of Southampton\.External Links:[Link](https://eprints.soton.ac.uk/467536/)
- \[34\]M\. L\. Puterman\(1994\)Markov decision processes: discrete stochastic dynamic programming\.John Wiley & Sons\.Note:Reprinted 2014Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p1.1)\.
- \[35\]N\. Rahaman, D\. Aristoff, F\. Draxler, L\. Aitchison, J\. Martens, and A\. Lacoste\(2020\)Learning representations that support extrapolation\.arXiv preprint arXiv:2006\.16438\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p7.1),[§2\.3](https://arxiv.org/html/2609.27745#S2.SS3.p3.1)\.
- \[36\]D\. Silver, J\. Schrittwieser, K\. Simonyan, I\. Antonoglou, A\. Huang, A\. Guez, T\. Hubert, L\. Baker, M\. Lai, A\. Bolton,et al\.\(2017\)Mastering the game of Go without human knowledge\.Nature550\(7676\),pp\. 354–359\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p1.1)\.
- \[37\]R\. S\. Sutton and A\. G\. Barto\(2018\)Reinforcement learning: an introduction\.Second edition,MIT Press\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p1.1)\.
- \[38\]R\. S\. Sutton\(1988\)Learning to predict by the methods of temporal differences\.Machine Learning3\(1\),pp\. 9–44\.Cited by:[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p5.1)\.
- \[39\]M\. Temizöz, R\. Vanvuchelen, E\. Gijsbrechts, and R\. Boute\(2025\)Reinforcement learning in inventory control under uncertainty\.arXiv preprint arXiv:2503\.05276\.
- \[40\]E\. van der Pol, T\. Kierfeld, and M\. Welling\(2020\)MDP homomorphic networks: group symmetries in reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 4199–4210\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p3.1)\.
- \[41\]O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. Dudzik, J\. Chung, D\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev,et al\.\(2019\)Grandmaster level in StarCraft II using multi\-agent reinforcement learning\.Nature575\(7782\),pp\. 350–354\.
- \[42\]A\. Vistoli\(2005\)Grothendieck topologies, fibered categories and descent theory\.InFundamental Algebraic Geometry: Grothendieck’s FGA Explained,Mathematical Surveys and Monographs, Vol\.123,pp\. 1–104\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p3.1)\.
- \[43\]X\. Wang, Y\. Chen, Z\. Li, G\. Zhang, T\. Yang, and J\. Li\(2023\)Fantastic rewards and how to tame them: a case study on reward learning for task\-oriented dialogue systems\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,pp\. 11736–11749\.
- \[44\]Z\. Wang, M\. Zhang, X\. Hou, H\. Xu, C\. Xu, C\. Xu, and L\. Wang\(2022\)Equivariant Q learning in spatial action spaces\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 22804–22819\.Cited by:[§2\.1](https://arxiv.org/html/2609.27745#S2.SS1.p7.1)\.
- \[45\]C\. J\. C\. H\. Watkins and P\. Dayan\(1992\)Q\-learning\.Machine Learning8\(3\-4\),pp\. 279–292\.Cited by:[§3\.1](https://arxiv.org/html/2609.27745#S3.SS1.p2.1)\.
- \[46\]C\. J\. C\. H\. Watkins\(1989\)Learning from delayed rewards\.Ph\.D\. Dissertation,University of Cambridge\.
- \[47\]S\. Weinberg\(1995\)The quantum theory of fields, volume 1: foundations\.Cambridge University Press,Cambridge\.External Links:ISBN 978\-0521550017
- \[48\]A\. Weinstein\(1996\)Groupoids: unifying internal and external symmetry\.Notices of the AMS43\(7\),pp\. 744–752\.Cited by:[§1](https://arxiv.org/html/2609.27745#S1.p3.1),[§2\.5](https://arxiv.org/html/2609.27745#S2.SS5.p2.1)\.
- \[49\]D\. E\. Worrall, S\. J\. Garbin, D\. Turmukhambetov, and G\. J\. Brostow\(2017\)Harmonic networks: deep translation and rotation equivariance\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 5028–5037\.
- \[50\]C\. Zhang, D\. Wu, and J\. Zhu\(2020\)Interterminal truck routing optimization using deep reinforcement learning\.Sensors20\(20\),pp\. 5794\.External Links:[Document](https://dx.doi.org/10.3390/s20205794)
- \[51\]X\. Zhao, L\. Xia, Y\. Zhang, Z\. Ding, D\. Yin, and J\. Tang\(2019\)Deep reinforcement learning for search, recommendation, and online advertising: a survey\.SIGKDD Explorations Newsletter21\(2\),pp\. 25–47\.

\*