Factorized Spectral Representations for Reinforcement Learning

# Factorized Spectral Representations for Reinforcement Learning
Source: [https://arxiv.org/html/2607.13498](https://arxiv.org/html/2607.13498)
Junyi Wu Department of Industrial and Systems Engineering University of Washington Seattle, WA, USA junyiwu@uw\.edu&Dan Li Department of Industrial and Systems Engineering University of Washington Seattle, WA, USA dli27@uw\.edu

###### Abstract

Learning a compact model of the world from interaction data is central to sample\-efficient deep reinforcement learning\. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state\-action pairs on one side and next states on the other, and learning a low\-rank factorization through self\-supervised contrastive objectives\. We take this view one step further\. The transition kernel is naturally a three\-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode\. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next\-state encoders that together form a single spectral representation\. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions\. Empirically, FaStR delivers its largest gains on high\-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained\.

## 1Introduction

Sample efficiency is a central concern in continuous\-control RL\. As state and action dimensions grow, the number of environment interactions required by model\-free methods rises sharply, limiting real\-world deployment\(Fujimotoet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib2); Haarnojaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib14)\)\. One productive route is the framework of low\-rank MDPs\(Jinet al\.,[2020](https://arxiv.org/html/2607.13498#bib.bib15); Yang and Wang,[2020](https://arxiv.org/html/2607.13498#bib.bib26); Agarwalet al\.,[2020](https://arxiv.org/html/2607.13498#bib.bib27)\), which assumes the transition kernel admits a finite\-rank factorization\. The representation\-learning problem then becomes one of finding a feature space in which the dynamics are lix—near\. Spectral representation methods bring this low\-rank view closer to practical deep RL\(Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\)\. They use transition samples from online interaction to learn transition\-aware features, often without reward labels, and feed these features to policy and value learning\. When such features are estimated more accurately from the same interaction budget, the critic receives a representation closer to the Bellman\-linear structure earlier in training\. The contrastive objective used to extract these features has been refined steadily across this line of work, and the resulting agents are now competitive on standard benchmarks\.

In many control systems, states and actions play different roles\. The state describes the current configuration of the system, while the action describes the input driving the system\. Existing spectral RL methods\(Renet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib18); Zhanget al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib10); Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12); Shribaket al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib21); Nabatiet al\.,[2026](https://arxiv.org/html/2607.13498#bib.bib23)\)implement low\-rank structure by treating the state\-action pair as a joint argument to the transition kernel\. Separately, factored value\-function methods separate state and action structures within the critic\. This work identifies a gap between these two approaches: the lack of a factored low\-rank representation for the transition dynamics themselves\. The motivation for this connection stems from the observation that distinct actions often drive identical state\-dependent motion patterns\.

![Refer to caption](https://arxiv.org/html/2607.13498v1/x1.png)\(a\)Joint Representation
![Refer to caption](https://arxiv.org/html/2607.13498v1/x2.png)\(b\)Factorized Representation

Figure 1:Comparison of representation structures\. Factorization constrains the joint score landscape to the intersection of independent state and action profiles\.In such cases, a joint representation must relearn shared state\-side structure across actions, whereas a factored representation can reuse the same state\-side modes while learning how actions modulate them\.

To this end, we introduce Factored Spectral Representations \(FaStR\), which treats the transition kernel as a three\-way tensor and learns one encoder per mode \(state, action, next\-state\)\. Figure[1](https://arxiv.org/html/2607.13498#S1.F1)illustrates the key difference\. A joint encoder maps the concatenated pair\(s,a\)\(s,a\)directly into a state\-action representation, so the learned score can vary freely over the full joint plane \(Figure[1\(a\)](https://arxiv.org/html/2607.13498#S1.F1.sf1)\)\. In this view, local peaks in the score landscape may be tied to specific state\-action regions\. FaStR instead learns separate state and action profiles, and combines them through a Hadamard product \(Figure[1\(b\)](https://arxiv.org/html/2607.13498#S1.F1.sf2)\)\. Thus, peaks and variations in the joint score must arise from the interaction of reusable state\-side and action\-side factors, encouraging sharing across rows and columns of the state\-action plane\. This structured sharing acts as an implicit regularizer and shifts the statistical burden from covering isolated state\-action regions to estimating reusable factors, allowing the frozen feature to become useful for Bellman backups with fewer transitions when the dynamics match the factorization\. Specifically, our contributions are as follows:

1. 1\.A factored view of spectral RL representations\.We formulate the transition kernel as a three\-mode object over state, action, and next state, and instantiate this view as a representation with a separate encoder for each of these three modes\. The factored representation keeps these modes explicit while remaining compatible with contrastive training\.
2. 2\.A representation\-complexity separation under CP structure\.We prove that the factored hypothesis class has a covering number that decomposes additively over the state, action, and next\-state factors\. Under CP realizability, this gives a sample\-size gap implied by the bound; in the approximate case, the guarantee becomes a bias\-estimation tradeoff\.
3. 3\.Empirical evaluation and modular transfer\.On DM Control Suite\(Tassaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib11)\)tasks across morphologies and action dimensions, we test when the factored representation outperforms a joint state\-action encoder\. Gains grow with action dimension, but only when the factorization matches the dynamics\. The same factorization also supports modular adaptation: under actuator shift, the state factor transfers intact and only the action factor is retrained\.

## 2Preliminaries

We consider a discounted MDPℳ=\(𝒮,𝒜,P,r,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma\)with continuous state space𝒮\\mathcal\{S\}, continuous action space𝒜\\mathcal\{A\}, transition kernelP:𝒮×𝒜→Δ​\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\Delta\(\\mathcal\{S\}\), reward functionr:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}, and discount factorγ∈\[0,1\)\\gamma\\in\[0,1\)\. The Q\-functionQπ​\(s,a\)=𝔼π​\[∑t=0∞γt​r​\(st,at\)∣s0=s,a0=a\]Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}=s,a\_\{0\}=a\]satisfies the Bellman equation

Qπ​\(s,a\)=r​\(s,a\)\+γ​∫𝒮P​\(s′\|s,a\)​Vπ​\(s′\)​𝑑s′,Q^\{\\pi\}\(s,a\)=r\(s,a\)\+\\gamma\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\|s,a\)\\,V^\{\\pi\}\(s^\{\\prime\}\)\\,ds^\{\\prime\},\(1\)whereVπ​\(s\)=𝔼a∼π\(⋅\|s\)​\[Qπ​\(s,a\)\]V^\{\\pi\}\(s\)=\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}\[Q^\{\\pi\}\(s,a\)\]\.

##### Linear MDPs and the linear\-feature view\.

A common simplification is to assume the transition kernel admits a finite\-rank factorization\. The linear MDP assumption\(Jinet al\.,[2020](https://arxiv.org/html/2607.13498#bib.bib15)\)states that there exist a feature mapφ:𝒮×𝒜→ℝd\\varphi:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\}, a kernel mapμ:𝒮→ℝd\\mu:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\}, and a reward parameterθr∈ℝd\\theta\_\{r\}\\in\\mathbb\{R\}^\{d\}such that

P​\(s′\|s,a\)=φ​\(s,a\)⊤​μ​\(s′\),r​\(s,a\)=φ​\(s,a\)⊤​θr\.P\(s^\{\\prime\}\|s,a\)=\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\),\\qquad r\(s,a\)=\\varphi\(s,a\)^\{\\top\}\\theta\_\{r\}\.\(2\)Under this assumption, the value function is linear inφ\\varphi: for any policyπ\\pi,Qπ​\(s,a\)=φ​\(s,a\)⊤​wπQ^\{\\pi\}\(s,a\)=\\varphi\(s,a\)^\{\\top\}w^\{\\pi\}for somewπ∈ℝdw^\{\\pi\}\\in\\mathbb\{R\}^\{d\}that absorbs the integral againstVπV^\{\\pi\}\. The complexity of value learning then depends on the feature dimensionddrather than the size of the state\-action space, which enables provably efficient algorithms\(Jinet al\.,[2020](https://arxiv.org/html/2607.13498#bib.bib15); Ueharaet al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib28)\)\. In practiceφ\\varphiis unknown and must be learned from interaction data\.

##### Spectral RL: learning transition\-ratio scores\.

Spectral representation methods learnφ\\varphiandμ\\mufrom transition samples using self\-supervised contrastive objectives\(Gutmann and Hyvärinen,[2010](https://arxiv.org/html/2607.13498#bib.bib49); van den Oordet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib50); Zhanget al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib10); Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\)\. A scoring functionh​\(s,a,s′\)=φ​\(s,a\)⊤​μ​\(s′\)h\(s,a,s^\{\\prime\}\)=\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\)is trained to assign high values to true next\-state sampless′∼P\(⋅\|s,a\)s^\{\\prime\}\\sim P\(\\cdot\|s,a\)and low values to negatives drawn from a reference distribution\. At the population level, such objectives fit a transition\-ratio score: the logit measures how plausible a candidate next state is under the transition kernel relative to the negative distribution\. Existing deep spectral methods parameterize this score bilinearly, using a fused state\-action feature on one side and a next\-state feature on the other\. The trainedφ\\varphican then be used as the linear\-MDP feature for downstream value learning\.

The contrastive template that FaStR will instantiate is the ranking\-based perturbed noise contrastive estimation \(RP\-NCE\) objective\(Zhanget al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib10); Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\), in which negatives are obtained by perturbing true next states across a sequence of noise levels and the model is trained to rank a true \(perturbed\) positive aboveK−1K\-1corrupted alternatives:

ℓNCE=−1M​N​∑m=1M∑i=1Nlog⁡exp⁡\(φi⊤​μ​\(s~i,0′;βm\)\)∑j=0K−1exp⁡\(φi⊤​μ​\(s~i,j′;βm\)\),\\ell\_\{\\text\{NCE\}\}=\-\\frac\{1\}\{MN\}\\sum\_\{m=1\}^\{M\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{\\exp\\bigl\(\\varphi\_\{i\}^\{\\top\}\\mu\(\\tilde\{s\}^\{\\prime\}\_\{i,0\};\\beta\_\{m\}\)\\bigr\)\}\{\\sum\_\{j=0\}^\{K\-1\}\\exp\\bigl\(\\varphi\_\{i\}^\{\\top\}\\mu\(\\tilde\{s\}^\{\\prime\}\_\{i,j\};\\beta\_\{m\}\)\\bigr\)\},\(3\)whereφi=φ​\(si,ai\)\\varphi\_\{i\}=\\varphi\(s\_\{i\},a\_\{i\}\),\{βm\}m=1M\\\{\\beta\_\{m\}\\\}\_\{m=1\}^\{M\}are noise levels along a variance\-preserving \(VP\) schedule\(Songet al\.,[2021](https://arxiv.org/html/2607.13498#bib.bib54)\), which keeps the marginal variance fixed across levels, ands~i,j′\\tilde\{s\}^\{\\prime\}\_\{i,j\}denotes thejj\-th candidate next state corrupted at levelβm\\beta\_\{m\}, withj=0j=0the true target andj≥1j\\geq 1negatives drawn from other transitions in the mini\-batch \(in\-batch negatives\)\. At the population level this ranking objective is consistent with maximum likelihood estimation as the number of negatives grows\(Ma and Collins,[2018](https://arxiv.org/html/2607.13498#bib.bib29)\)\. Thus RP\-NCE is best viewed as an estimator for a chosen transition\-ratio score class; FaStR keeps this estimator but changes the score class from a fused bilinear to a mode\-wise trilinear form\.

##### The full spectral RL pipeline\.

The contrastive loss is one component of the system\. After training, the encoderφ\\varphiis used as a frozen representation: a criticQθQ\_\{\\theta\}takesφ​\(s,a\)\\varphi\(s,a\)as input and is trained by standard temporal\-difference \(TD\) losses without back\-propagating throughφ\\varphi, and a policy networkπ\\piis trained on top of this frozen representation\. This separation between representation learning \(the contrastive loss\) and value learning \(TD on the frozen feature\) is what makes the learnedφ\\varphiusable as the linear\-MDP feature inJinet al\.\([2020](https://arxiv.org/html/2607.13498#bib.bib15)\)\. We work within this two\-stage setup and locate our contribution in the structural form of the encoderφ\\varphi\.

## 3Method

### 3\.1The FaStR Framework

##### A tensor view of the transition kernel\.

The standard linear MDP mergesssandaainto a single index and factorizes the resulting matrixP∈ℝ\|𝒮×𝒜\|×\|𝒮\|P\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\\times\\mathcal\{A\}\|\\times\|\\mathcal\{S\}\|\}via an SVD\-like decomposition\. This treats the two arguments uniformly, although the state specifies the configuration of the system while the action specifies the input applied to it\. We separate the two arguments by viewing the transition kernel as a third\-way tensor𝒯∈ℝ\|𝒮\|×\|𝒜\|×\|𝒮\|\\mathcal\{T\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\\times\|\\mathcal\{A\}\|\\times\|\\mathcal\{S\}\|\}indexed byss,aa, ands′s^\{\\prime\}\.111The tensor view is stated in the finite case for intuition; Assumption[1](https://arxiv.org/html/2607.13498#Thmassumption1)below is stated directly through feature maps on general𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}\.

A standard general decomposition of such a tensor is the Tucker form\(Tucker,[1966](https://arxiv.org/html/2607.13498#bib.bib46); Kolda and Bader,[2009](https://arxiv.org/html/2607.13498#bib.bib45)\)P​\(s′\|s,a\)=∑i,j,kGi​j​k​ϕs,i​\(s\)​ϕa,j​\(a\)​mk​\(s′\)P\(s^\{\\prime\}\|s,a\)=\\sum\_\{i,j,k\}G\_\{ijk\}\\,\\phi\_\{s,i\}\(s\)\\,\\phi\_\{a,j\}\(a\)\\,m\_\{k\}\(s^\{\\prime\}\)with a coreG∈ℝds×da×dmG\\in\\mathbb\{R\}^\{d\_\{s\}\\times d\_\{a\}\\times d\_\{m\}\}\. The CP \(CANDECOMP/PARAFAC\) decomposition\(Carroll and Chang,[1970](https://arxiv.org/html/2607.13498#bib.bib47); Harshman,[1970](https://arxiv.org/html/2607.13498#bib.bib48); Kolda and Bader,[2009](https://arxiv.org/html/2607.13498#bib.bib45)\)is the special case whereGGis superdiagonal,Gk​k​k=1G\_\{kkk\}=1, eliminating the core entirely; lower\-rank or block\-diagonal cores interpolate between CP and dense Tucker\. Among these, FaStR uses the CP form; the case for this choice is made below in the context of contrastive training\.

###### Assumption 1\(CP\-Factored MDP\)\.

There exist feature mapsϕs:𝒮→ℝd\\phi\_\{s\}:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\},ϕa:𝒜→ℝd\\phi\_\{a\}:\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\}, a kernel mapm:𝒮→ℝdm:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\}, and a reward parameterθr∈ℝd\\theta\_\{r\}\\in\\mathbb\{R\}^\{d\}such that for all\(s,a,s′\)\(s,a,s^\{\\prime\}\):

P​\(s′\|s,a\)\\displaystyle P\(s^\{\\prime\}\|s,a\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)=∑k=1dϕs,k​\(s\)​ϕa,k​\(a\)​mk​\(s′\),\\displaystyle=\\bigl\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\\bigr\)^\{\\top\}m\(s^\{\\prime\}\)=\\sum\_\{k=1\}^\{d\}\\phi\_\{s,k\}\(s\)\\,\\phi\_\{a,k\}\(a\)\\,m\_\{k\}\(s^\{\\prime\}\),\(4\)r​\(s,a\)\\displaystyle r\(s,a\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​θr\.\\displaystyle=\\bigl\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\\bigr\)^\{\\top\}\\theta\_\{r\}\.\(5\)We writeψ​\(s,a\):=ϕs​\(s\)⊙ϕa​\(a\)∈ℝd\\psi\(s,a\):=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\\in\\mathbb\{R\}^\{d\}for the factored feature\.

Each componentkkin Equation ref\{eq:cp\} is a rank\-one term combining a state factorϕs,k​\(s\)\\phi\_\{s,k\}\(s\), an action factorϕa,k​\(a\)\\phi\_\{a,k\}\(a\), and a next\-state factormk​\(s′\)m\_\{k\}\\left\(s^\{\\prime\}\\right\), with the action factor scaling the coupling between the other two\. CP fits well when actions scale shared state\-dependent modes of motion, such as posture or balance in locomotion; it underfits when strong contact or limb\-coupling effects entangle modes across actions\. The CP form has scale invariances betweenϕs\\phi\_\{s\},ϕa\\phi\_\{a\}, andmm, which are fixed by uniform norm boundsBϕB\_\{\\phi\}andBm,∞B\_\{m,\\infty\}throughout the analysis \(Appendix[C\.1](https://arxiv.org/html/2607.13498#A3.SS1)\); the implementation uses LayerNorm to stabilize the scales\.

##### Factored representation and Q\-linearity\.

FaStR learns a feature that is useful for control, not only for predicting the next state\. Letψ​\(s,a\):=ϕs​\(s\)⊙ϕa​\(a\)\\psi\(s,a\):=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\. Under the CP\-factored model, the transition and reward share this same feature:P​\(s′\|s,a\)=ψ​\(s,a\)⊤​m​\(s′\)P\(s^\{\\prime\}\|s,a\)=\\psi\(s,a\)^\{\\top\}m\(s^\{\\prime\}\)andr​\(s,a\)=ψ​\(s,a\)⊤​θrr\(s,a\)=\\psi\(s,a\)^\{\\top\}\\theta\_\{r\}\. Therefore, a Bellman backup does not leave the span ofψ\\psi: for any policyπ\\pi,Qπ​\(s,a\)Q^\{\\pi\}\(s,a\)is derived as:

r\(s,a\)\+γ∫𝒮P\(s′\|s,a\)Vπ\(s′\)ds′=ψ\(s,a\)⊤\(θr\+γ∫𝒮m\(s′\)Vπ\(s′\)ds′\)=:ψ\(s,a\)⊤wπ\.r\(s,a\)\+\\gamma\\\!\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\|s,a\)V^\{\\pi\}\(s^\{\\prime\}\)ds^\{\\prime\}=\\psi\(s,a\)^\{\\top\}\\\!\\left\(\\theta\_\{r\}\+\\gamma\\\!\\int\_\{\\mathcal\{S\}\}m\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)ds^\{\\prime\}\\right\)=:\\psi\(s,a\)^\{\\top\}w^\{\\pi\}\.\(6\)Thus the downstream linear critic is not an additional assumption: once the transition and reward admit the CP form, the correspondingQQ\-function is linear in the learned factored feature\. A derivation with norm bounds is given in Appendix[C\.2](https://arxiv.org/html/2607.13498#A3.SS2)\.

##### Learning a trilinear transition\-ratio score\.

We learn the factors by viewing contrastive spectral learning as conditional ratio estimation\. For an anchor\(si,ai\)\(s\_\{i\},a\_\{i\}\), the observed next state is the positive candidate and perturbed in\-batch next states are negatives\. InfoNCE\(van den Oordet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib50)\)turns this into a classification problem: among one true next state and several distractors, the model is trained to assign the highest energy to the true candidate\. At the population optimum, differences between these energies estimate the log ratio between the transition law at the anchor and the negative candidate distribution\. RP\-NCE repeats this classification across noise levelsβ\\beta, giving a multi\-scale ratio target for the next\-state side\. In standard spectral RL, this energy is bilinear: a fused state\-action feature is matched with aβ\\beta\-conditioned next\-state feature\. FaStR instead chooses the CP trilinear energy

fΘ​\(s,a,s′;β\)=∑k=1dϕs,Θ,k​\(s\)​ϕa,Θ,k​\(a\)​mΘ,k​\(s′;β\)=\(ϕs,Θ​\(s\)⊙ϕa,Θ​\(a\)\)⊤​mΘ​\(s′;β\)\.f\_\{\\Theta\}\(s,a,s^\{\\prime\};\\beta\)=\\sum\_\{k=1\}^\{d\}\\phi\_\{s,\\Theta,k\}\(s\)\\,\\phi\_\{a,\\Theta,k\}\(a\)\\,m\_\{\\Theta,k\}\(s^\{\\prime\};\\beta\)=\\bigl\(\\phi\_\{s,\\Theta\}\(s\)\\odot\\phi\_\{a,\\Theta\}\(a\)\\bigr\)^\{\\top\}m\_\{\\Theta\}\(s^\{\\prime\};\\beta\)\.Each coordinate is a ratio channel: the state factor selects a state\-dependent mode, the action factor gates it, and the next\-state factor scores the candidate at noise levelβ\\beta\. Substituting this energy into the RP\-NCE template \([3](https://arxiv.org/html/2607.13498#S2.E3)\) gives

ℓFaStR=−1M​N​∑b=1M∑i=1Nlog⁡exp⁡\(fΘ​\(si,ai,s~i,0′;βb\)\)∑j=0K−1exp⁡\(fΘ​\(si,ai,s~i,j′;βb\)\)\.\\ell\_\{\\mathrm\{FaStR\}\}=\-\\frac\{1\}\{MN\}\\sum\_\{b=1\}^\{M\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{\\exp\\\!\\left\(f\_\{\\Theta\}\(s\_\{i\},a\_\{i\},\\tilde\{s\}^\{\\prime\}\_\{i,0\};\\beta\_\{b\}\)\\right\)\}\{\\sum\_\{j=0\}^\{K\-1\}\\exp\\\!\\left\(f\_\{\\Theta\}\(s\_\{i\},a\_\{i\},\\tilde\{s\}^\{\\prime\}\_\{i,j\};\\beta\_\{b\}\)\\right\)\}\.\(7\)The learned state\-action feature is the anchor side of this energy,ψΘ​\(s,a\)=ϕs,Θ​\(s\)⊙ϕa,Θ​\(a\)\\psi\_\{\\Theta\}\(s,a\)=\\phi\_\{s,\\Theta\}\(s\)\\odot\\phi\_\{a,\\Theta\}\(a\)\. The downstream RFF critic then kernelizes the frozenψΘ\\psi\_\{\\Theta\}for value learning, while the mode\-wise geometry ofψΘ\\psi\_\{\\Theta\}is fixed by the transition\-ratio energy\. Together, the preceding pieces define the full FaStR pipeline: the CP factored MDP specifies the trilinear score class, Q\-linearity explains why the anchor feature can support value learning, and RP\-NCE provides the self\-supervised signal that learns the factors from transitions\. After this representation stage, we freezeψΘ\\psi\_\{\\Theta\}, train the actor and critic on top of it, and use the same downstream control pipeline as prior spectral RL; the structural change is in how transition data shape the state\-action feature\. Algorithm[1](https://arxiv.org/html/2607.13498#alg1)gives the per\-step update\.

##### CP vs\. Tucker cores\.

The CP form is the diagonal member of a broader trilinear transition\-ratio score class, written ash​\(s,a,s′\)=ϕs​\(s\)⊤​M​\(s′\)​ϕa​\(a\)h\(s,a,s^\{\\prime\}\)=\\phi\_\{s\}\(s\)^\{\\top\}M\(s^\{\\prime\}\)\\phi\_\{a\}\(a\), withM​\(s′\)M\(s^\{\\prime\}\)generated by the next\-state mode and restricted to be diagonal under CP\. Other parameterizations correspond to less constrained interaction matrices: a denseds×dad\_\{s\}\\times d\_\{a\}matrix gives full Tucker, and intermediate cases \(low\-rankM​\(s′\)=U​\(s′\)​V​\(s′\)⊤M\(s^\{\\prime\}\)=U\(s^\{\\prime\}\)V\(s^\{\\prime\}\)^\{\\top\}, block\-diagonal cores\) interpolate between CP and dense Tucker\. Such factorizations have been used in prior bilinear MDP methods trained with regression or maximum\-likelihood objectives\(Yang and Wang,[2020](https://arxiv.org/html/2607.13498#bib.bib26); Duet al\.,[2021](https://arxiv.org/html/2607.13498#bib.bib37); Linet al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib36)\), where the gradient onM​\(s′\)M\(s^\{\\prime\}\)is full\-rank per anchor and large interaction matrices can be trained with enough data\.

Under contrastive training the gradient structure is different\. The chain rule onhi​j=ϕs​\(si\)⊤​M​\(sj′\)​ϕa​\(ai\)h\_\{ij\}=\\phi\_\{s\}\(s\_\{i\}\)^\{\\top\}M\(s^\{\\prime\}\_\{j\}\)\\phi\_\{a\}\(a\_\{i\}\)gives

∇vec​\(M​\(sj′\)\)ℓNCE∈span​\{ϕa​\(ai\)⊗ϕs​\(si\)\},\\nabla\_\{\\mathrm\{vec\}\(M\(s^\{\\prime\}\_\{j\}\)\)\}\\,\\ell\_\{\\text\{NCE\}\}\\;\\in\\;\\mathrm\{span\}\\bigl\\\{\\phi\_\{a\}\(a\_\{i\}\)\\otimes\\phi\_\{s\}\(s\_\{i\}\)\\bigr\\\},\(8\)a one\-dimensional subspace per anchor\. A mini\-batch ofNNanchors therefore covers at mostmin⁡\(N,dim\(M\)\)\\min\(N,\\dim\(M\)\)directions in the parameter space ofM​\(s′\)M\(s^\{\\prime\}\)in a single update\. This rank\-one structure follows from the conditional bilinear form of the trilinear score and is the same for any parameterization ofMM: low\-rank or block\-diagonal cores reduce the parameter count belowds​dad\_\{s\}d\_\{a\}, but each anchor still contributes a rank\-one direction projected onto the corresponding submanifold\. For dense Tucker,dim\(M\)=ds​da\\dim\(M\)=d\_\{s\}d\_\{a\}, so per\-batch coverage requiresN≥ds​daN\\geq d\_\{s\}d\_\{a\}, which scales quadratically in the per\-mode dimension and exceeds typical batch sizes\. The CP form hasdim\(M\)=d\\dim\(M\)=d, so the linear requirementN≥dN\\geq dis met by standard mini\-batch sizes\.

Whether structured cores can accumulate the full parameter space across many SGD updates at realistic compute is an empirical question\. The underlying issue is that any structured\-core fix faces a three\-way constraint: encoder capacity, NCE validity \(the negative count must reach the effective NCE dimension, hereN≥ds​daN\\geq d\_\{s\}d\_\{a\}, for the contrastive logit to be non\-degenerate\), and per\-batch gradient coverage cannot be satisfied simultaneously at standard batch sizes\. Reducingds,dad\_\{s\},d\_\{a\}to satisfy validity makes encoder capacity the binding constraint and removes the representational benefit\. Appendix[D\.1](https://arxiv.org/html/2607.13498#A4.SS1)reports a controlled three\-way encoder ablation \(monolithic, concat\-of\-separate, Tucker\) and a separate capacity test, showing that no choice of Tucker rank within this constraint matches CP under matched compute\.

### 3\.2Provable Representation Efficiency

Section[3\.1](https://arxiv.org/html/2607.13498#S3.SS1)chose CP among bilinear parameterizations on training\-dynamics grounds\. The matching statistical question is whether CP also makes the representation class easier to learn from a finite sample\. We answer it in three steps\. We first bound the size of the factored class relative to the joint class \(Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\)\. We then convert this bound into a finite\-sample bound on representation error \(Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)\)\. Finally, we invert the bound to compare the sample sizes that suffice for each class to reach a target error \(Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)\)\. The dimension dependence of the final result matches the empirical pattern in Section[4](https://arxiv.org/html/2607.13498#S4)\.

We work with two scoring classes\. The factored class𝒢fac\\mathcal\{G\}\_\{\\text\{fac\}\}containsg​\(s,a,s′\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)g\(s,a,s^\{\\prime\}\)=\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\)with\(ϕs,ϕa,m\)∈ℱs×ℱa×ℱm\(\\phi\_\{s\},\\phi\_\{a\},m\)\\in\\mathcal\{F\}\_\{s\}\\times\\mathcal\{F\}\_\{a\}\\times\\mathcal\{F\}\_\{m\}; the joint class𝒢joint\\mathcal\{G\}\_\{\\text\{joint\}\}containsg​\(s,a,s′\)=φ​\(s,a\)⊤​μ​\(s′\)g\(s,a,s^\{\\prime\}\)=\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\)withφ∈ℱs​a\\varphi\\in\\mathcal\{F\}\_\{sa\},μ∈ℱm\\mu\\in\\mathcal\{F\}\_\{m\}\. Norm bounds and the convention for sup\-ℓ2\\ell\_\{2\}covering numbers are in Appendix[C\.1](https://arxiv.org/html/2607.13498#A3.SS1)\.

##### Step 1: bounding the size of the factored class\.

A monolithic encoder takes each\(s,a\)\(s,a\)pair as a single input, and the size of its hypothesis class scales with the combined state\-action dimension\. The CP parameterization splits the score into three networks: a state factor, an action factor, and a kernel encoder, coupled through a diagonal trilinear interaction \(the Hadamard form of Eq\.[4](https://arxiv.org/html/2607.13498#S3.E4)\)\. The size of the resulting class is then controlled by the three component networks separately rather than by their product\. Covering numbers, a standard measure of the size of a function class, make this precise\.

###### Theorem 3\.1\(Covering\-number decomposition\)\.

For everyϵ\>0\\epsilon\>0,

log⁡𝒩​\(𝒢*fac*,ϵ\)≤log⁡𝒩​\(ℱs,ϵ3​Bm,∞​Bϕ\)\+log⁡𝒩​\(ℱa,ϵ3​Bm,∞​Bϕ\)\+log⁡𝒩​\(ℱm,ϵ3​Bϕ2\)\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\emph\{fac\}\},\\epsilon\)\\;\\leq\\;\\log\\mathcal\{N\}\\\!\\left\(\\mathcal\{F\}\_\{s\},\\tfrac\{\\epsilon\}\{3B\_\{m,\\infty\}B\_\{\\phi\}\}\\right\)\+\\log\\mathcal\{N\}\\\!\\left\(\\mathcal\{F\}\_\{a\},\\tfrac\{\\epsilon\}\{3B\_\{m,\\infty\}B\_\{\\phi\}\}\\right\)\+\\log\\mathcal\{N\}\\\!\\left\(\\mathcal\{F\}\_\{m\},\\tfrac\{\\epsilon\}\{3B\_\{\\phi\}^\{2\}\}\\right\)\.Ifℱs​a\\mathcal\{F\}\_\{sa\}contains every Hadamard productϕs⊙ϕa\\phi\_\{s\}\\odot\\phi\_\{a\}with\(ϕs,ϕa\)∈ℱs×ℱa\(\\phi\_\{s\},\\phi\_\{a\}\)\\in\\mathcal\{F\}\_\{s\}\\times\\mathcal\{F\}\_\{a\}, then𝒢*fac*⊆𝒢*joint*\\mathcal\{G\}\_\{\\emph\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\emph\{joint\}\}, and the inclusion holds for both covering numbers and Rademacher complexities at every sample size\.

###### Proof\.

See Appendix[C\.3](https://arxiv.org/html/2607.13498#A3.SS3)\. ∎

For Lipschitz encoders on compact domains, this gives an exponentDmax=max⁡\(ds,da\)D\_\{\\max\}=\\max\(d\_\{s\},d\_\{a\}\)for the factored class versusDsum=ds\+daD\_\{\\mathrm\{sum\}\}=d\_\{s\}\+d\_\{a\}for the joint class, leaving a gap ofmin⁡\(ds,da\)\\min\(d\_\{s\},d\_\{a\}\)in the rate exponent that is formalized as a sample\-size separation in Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)\.

##### What the bound says in operational terms\.

In sample\-size terms, the covering number is the number of transitions needed for the contrastive loss to separate two encoder triples within a target tolerance\. A smaller covering number means the empirical loss is close to its population value at smallernn, so the resultingψ=ϕs⊙ϕa\\psi=\\phi\_\{s\}\\odot\\phi\_\{a\}is available as a TD feature at smallernn\. Take humanoid as a concrete case\. The state hasds=67d\_\{s\}=67proprioceptive coordinates, the action hasda=21d\_\{a\}=21actuator commands, and the standard approach fits the transition density on the joint8888\-dimensional input\. The CP form fits two smaller pieces instead: a6767\-dimensional state feature, and2121per\-actuator gains that say how each actuator scales that feature\. Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)states this same split at the level of the covering number\. Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)turns the smaller class size into a smaller representation error at fixednn\. Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)inverts the bound and gives the smaller sample size needed for a target error\. Onceψ\\psiis frozen, TD onψ\\psisatisfies the linear\-MDP sample\-complexity guarantees ofJinet al\.\([2020](https://arxiv.org/html/2607.13498#bib.bib15)\), so a smaller representation error at a givennnbecomes a smaller return gap in policy learning\.

##### Step 2: from class size to representation error\.

A smaller class size matters only if it leads to a smaller generalization gap\. The target we analyze is theL2L\_\{2\}representation objective ofRenet al\.\([2023](https://arxiv.org/html/2607.13498#bib.bib18)\), the squaredL2​\(ν\)L^\{2\}\(\\nu\)distance from the true transition density,

L​\(g\):=𝔼ρ​\[∫𝒮\(P​\(s′∣s,a\)−g​\(s,a,s′\)\)2​𝑑ν​\(s′\)\]\.L\(g\):=\\mathbb\{E\}\_\{\\rho\}\\\!\\left\[\\int\_\{\\mathcal\{S\}\}\\bigl\(P\(s^\{\\prime\}\\mid s,a\)\-g\(s,a,s^\{\\prime\}\)\\bigr\)^\{2\}d\\nu\(s^\{\\prime\}\)\\right\]\.We analyze this objective rather than NCE itself because finite\-sample guarantees are tractable for theL2L\_\{2\}surrogate at the population level, while the contrastive loss serves as its practical proxy at training time\(Renet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib18)\)\.

###### Proposition 3\.2\(Finite\-sample generalization\)\.

Let𝒢∈\{𝒢*fac*,𝒢*joint*\}\\mathcal\{G\}\\in\\\{\\mathcal\{G\}\_\{\\emph\{fac\}\},\\mathcal\{G\}\_\{\\emph\{joint\}\}\\\}with\|g\|≤Bg\|g\|\\leq B\_\{g\}, and letg^n\\hat\{g\}\_\{n\}be an empirical minimizer of the centered objective onnni\.i\.d\. transitions\. With probability at least1−δ1\-\\delta,

L​\(g^n\)≤L​\(g𝒢∗\)⏟approximation\+8​ℛn​\(𝒢\)\+4​ℛn​\(ℋ𝒢\)⏟estimation\+O​\(\(Bg\+Bg2\)​log⁡\(1/δ\)n\),L\(\\hat\{g\}\_\{n\}\)\\;\\leq\\;\\underbrace\{L\(g^\{\*\}\_\{\\mathcal\{G\}\}\)\}\_\{\\text\{approximation\}\}\\;\+\\;\\underbrace\{8\\,\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)\+4\\,\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)\}\_\{\\text\{estimation\}\}\\;\+\\;O\\\!\\left\(\(B\_\{g\}\+B\_\{g\}^\{2\}\)\\sqrt\{\\tfrac\{\\log\(1/\\delta\)\}\{n\}\}\\right\),whereℋ𝒢:=\{\(s,a\)↦∫g2​\(s,a,u\)​𝑑ν​\(u\):g∈𝒢\}\\mathcal\{H\}\_\{\\mathcal\{G\}\}:=\\\{\(s,a\)\\mapsto\\int g^\{2\}\(s,a,u\)\\,d\\nu\(u\):g\\in\\mathcal\{G\}\\\}is the integrated quadratic class arising from expanding the squared loss\. The quadratic class satisfies𝒩​\(ℋ𝒢,ϵ\)≤𝒩​\(𝒢,ϵ/\(2​Bg\)\)\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\},\\epsilon\)\\leq\\mathcal\{N\}\(\\mathcal\{G\},\\epsilon/\(2B\_\{g\}\)\), so its Rademacher complexity inherits the rate of𝒢\\mathcal\{G\}itself\.

###### Proof\.

See Appendix[C\.6](https://arxiv.org/html/2607.13498#A3.SS6)\. ∎

The bound has two terms with opposite scaling\. The approximation termL​\(g𝒢∗\)L\(g^\{\*\}\_\{\\mathcal\{G\}\}\)measures whether the class contains the true dynamics; under exact CP realizability and the inclusion of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1), both classes contain the truth and this term is zero for both\. Under approximate CP, the factored class has an irreducible biasϵCP2:=infg∈𝒢facL​\(g\)\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}:=\\inf\_\{g\\in\\mathcal\{G\}\_\{\\text\{fac\}\}\}L\(g\)that the joint class does not have\. The estimation term is smaller for the factored class: both Rademacher terms inherit the smaller exponentDmaxD\_\{\\max\}from Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1), so the factored class converges faster innn\.

##### Step 3: from representation error to sample size\.

The previous two steps bound representation error at a fixed sample size\. The inverse question is how many samples each class needs to reach the same target error\.

###### Proposition 3\.3\(Sufficient\-sample\-size separation\)\.

Assume exact CP realizability, the inclusion condition of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1), andLL\-Lipschitz encoder classes on compact domains withDmax≥3D\_\{\\max\}\\geq 3\. Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)then gives certificates of the formL​\(g^nfac\)≤Cfac​\(δ\)​n−1/DmaxL\(\\hat\{g\}\_\{n\}^\{\\mathrm\{fac\}\}\)\\leq C\_\{\\mathrm\{fac\}\}\(\\delta\)\\,n^\{\-1/D\_\{\\max\}\}andL​\(g^njoint\)≤Cjoint​\(δ\)​n−1/DsumL\(\\hat\{g\}\_\{n\}^\{\\mathrm\{joint\}\}\)\\leq C\_\{\\mathrm\{joint\}\}\(\\delta\)\\,n^\{\-1/D\_\{\\mathrm\{sum\}\}\}, withCfac,CjointC\_\{\\mathrm\{fac\}\},C\_\{\\mathrm\{joint\}\}depending polynomially on the norm bounds, Lipschitz constants, and feature dimension, and logarithmically on1/δ1/\\delta\. Letnfacsuff​\(ϵ\)n\_\{\\mathrm\{fac\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)andnjointsuff​\(ϵ\)n\_\{\\mathrm\{joint\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)be the smallestnnat which each certificate guarantees error at mostϵ2\\epsilon^\{2\}\. Then

njointsuff​\(ϵ\)nfacsuff​\(ϵ\)=Θ​\(ϵ−2​min⁡\(ds,da\)\)as​ϵ→0\.\\frac\{n\_\{\\mathrm\{joint\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}\{n\_\{\\mathrm\{fac\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}\\;=\\;\\Theta\\\!\\left\(\\epsilon^\{\-2\\min\(d\_\{s\},d\_\{a\}\)\}\\right\)\\quad\\text\{as \}\\epsilon\\to 0\.

###### Proof\.

See Appendix[C\.7](https://arxiv.org/html/2607.13498#A3.SS7)\. ∎

The factored class therefore has a smaller certified sample requirement, with the exponent gap equal tomin⁡\(ds,da\)\\min\(d\_\{s\},d\_\{a\}\)\. This gap is meaningful when the smaller of the state and action dimensions is nontrivial and the CP bias is small; otherwise the approximate\-CP certificateL​\(g^nfac\)≲ϵCP2\+Cfac​\(δ\)​n−1/DmaxL\(\\hat\{g\}\_\{n\}^\{\\mathrm\{fac\}\}\)\\lesssim\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}\+C\_\{\\mathrm\{fac\}\}\(\\delta\)n^\{\-1/D\_\{\\max\}\}can be dominated by bias\. We test these two conditions in Section[4](https://arxiv.org/html/2607.13498#S4)and Appendix[D\.2](https://arxiv.org/html/2607.13498#A4.SS2)\. Appendix[C\.8](https://arxiv.org/html/2607.13498#A3.SS8)further gives a conditional LSVI\-UCB implication: if the learned feature satisfies approximate Bellman linearity, the factored structure can reduce the representation cost of obtaining features suitable for optimism\-driven exploration\.

## 4Experiments

The experiments evaluate FaStR on two axes\. First, on standard locomotion benchmarks whose morphology is well aligned with a CP\-style factorization, we compare FaStR against the monolithic encoder under an otherwise identical pipeline \(Section[4\.2](https://arxiv.org/html/2607.13498#S4.SS2)\)\. Second, in a transfer setting where only the action\-to\-torque mapping changes between training and adaptation, we test whether the separable state factor enables sample\-efficient adaptation that a joint encoder cannot support \(Section[4\.3](https://arxiv.org/html/2607.13498#S4.SS3)\)\.

### 4\.1Setup

We evaluate on DM Control Suite\(Tassaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib11)\)locomotion tasks that vary in action dimensionality and morphology, withdim\(𝒜\)∈\{6,12,21,38\}\\dim\(\\mathcal\{A\}\)\\in\\\{6,12,21,38\\\}and shared proprioceptive observations\. To attribute performance differences to the encoder structure alone, we use a controlled protocol: FaStR and the monolithic baseline \(CTRL\-SR\(Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\)\) share identical total parameter counts, hidden dimensions, learning rates, noise schedules, and training budgets, with no task\-specific tuning for either method\. The only varying component is the representation: FaStR’s factored encoders\(ϕs,ϕa,m\)\(\\phi\_\{s\},\\phi\_\{a\},m\)versus CTRL\-SR’s joint encoderφ​\(\[s;a\]\)\\varphi\(\[s;a\]\)\. Main\-text learning curves \(Figure[2](https://arxiv.org/html/2607.13498#S4.F2)\) compare FaStR against CTRL\-SR\. Table[1](https://arxiv.org/html/2607.13498#S4.T1)additionally reports final\-return comparisons against Diff\-SR\(Shribaket al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib21)\), a second spectral RL baseline that replaces the contrastive objective with a diffusion next\-state model on the same TD3 backbone, and against TD7\(Fujimotoet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib41)\)and SAC\(Haarnojaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib14)\), the current model\-free SOTA\. SAC and TD7 use the architectures and hyperparameters from the original papers\. We report two SAC variants, SAC\(3\-layer\) and SAC\(2\-layer\), both wider per layer than the spectral methods’ actor, so a capacity mismatch is not the source of the gap\. We compare only against model\-free baselines; model\-based methods on this suite were evaluated byGaoet al\.\([2025](https://arxiv.org/html/2607.13498#bib.bib12)\)and gave lower returns than the spectral RL baselines\. Full hyperparameters are in Appendix[B](https://arxiv.org/html/2607.13498#A2)\.

### 4\.2Sample Efficiency on Locomotion Benchmarks

![Refer to caption](https://arxiv.org/html/2607.13498v1/x3.png)Figure 2:Learning curves across 8 DM Control Suite tasks spanning four morphologies\. FaStR’s gains are largest on tasks whose dynamics admit a CP factorization, consistent with the bias\-estimation view\. Shaded:±1\\pm 1std across 5 seeds\.On the Humanoid tasks, where torso\-limb morphology admits a CP factorization \(independent joint groups scaling shared postural modes\), FaStR gives higher returns than the controlled baseline by margins above seed\-level noise across all three tasks, attributing the difference to the representation\. On the Dog environment, the larger action dimension is partly offset by stronger ground\-contact dynamics that introduce state\-action coupling, giving improvements with wider variance\. The same ordering holds against Diff\-SR on all five high\-action\-dim tasks \(Table[1](https://arxiv.org/html/2607.13498#S4.T1)\), so the gain is not specific to the contrastive objective\.

On the Quadruped tasks, FaStR converges to CTRL\-SR’s terminal performance with higher sample efficiency: on Quadruped\-Run, it reaches the baseline’s converged level earlier in training and exceeds it asymptotically; on Quadruped\-Walk, the final return is slightly below the baseline, but FaStR reaches near\-converged performance earlier than CTRL\-SR at any fixed performance threshold\. The improvement on this family appears as faster convergence rather than a higher asymptote, which is consistent with the bias\-estimation interpretation: the estimation advantage appears in the rate at which the representation becomes usable for value learning\. On Cheetah\-Run, Walker\-Walk, and Walker\-Run \(dim𝒜≤6\\dim\\mathcal\{A\}\{\\leq\}6\), FaStR matches the baselines without significant gain, the regime predicted by Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)in which smallmin⁡\(ds,da\)\\min\(d\_\{s\},d\_\{a\}\)gives a small estimation advantage in the certificate\.

Table 1:Final\-10\-evaluation mean return on DM Control Suite tasks with proprioceptive observations, averaged across 5 seeds \(±1\\pm 1std\)\. Bold: best result where the gap exceeds one standard deviation\.##### Mechanism check\.

Appendix[D\.1](https://arxiv.org/html/2607.13498#A4.SS1)runs a controlled three\-way ablation that isolates the CP interaction from encoder separation and from generic multiplicative coupling\. A concat\-of\-separate variant, with independent encoders feeding a joint MLP, gives only a small improvement over the monolithic baseline, well below FaStR’s gain\. A full Tucker\-core variant has the rank\-deficient gradient issue of Section[3\.1](https://arxiv.org/html/2607.13498#S3.SS1), and a separate capacity test rules out the low\-rank workaround\. The CP interaction is therefore the source of the gain\. The offline CP\-misfit diagnostic \(Appendix[D\.2](https://arxiv.org/html/2607.13498#A4.SS2)\) further shows that gains track structural alignment with the dynamics, not action dimension alone\.

### 4\.3Modular Reuse ofϕs\\phi\_\{s\}under Actuator Shift

![Refer to caption](https://arxiv.org/html/2607.13498v1/x4.png)Figure 3:Architectural diagnostic under actuator shift\. Top: action permutation; bottom: actuator degradation\. Walker \(dim𝒜=6\\dim\\mathcal\{A\}\{=\}6\) is a pre\-shift parity check; Humanoid \(dim𝒜=21\\dim\\mathcal\{A\}\{=\}21\) gives the diagnostic\.What is preserved\(red solid vs\. blue dashed\): under the same freeze\-and\-adapt protocol, FaStR’sϕs\\phi\_\{s\}remains usable across the shift while CTRL\-SR’sφ\\varphidoes not\.At what cost\(red solid vs\. blue solid\): on Humanoid, FaStR with 18% of its parameters held fixed inϕs\\phi\_\{s\}reaches the same final return as CTRL\-SR retrained from scratch with all parameters trainable; trainable parameter counts for each condition are reported in the legend\. Orange dashed \(FaStR retrained from scratch\) is a reference upper bound\. Shaded:±1\\pm 1std across 5 seeds\. See Appendix[B](https://arxiv.org/html/2607.13498#A2)for parameter setting\.We test transfer in a setting where the physical dynamics of the system are unchanged but the action\-to\-torque interface is rewritten\. The CP factorization predicts what each architecture preserves under such a change\. Under FaStR,ϕs\\phi\_\{s\}depends only on the state\-side dynamics of the body and should remain valid when only the action\-to\-torque map shifts; under a joint encoder, state and action are coupled through every parameter, so any change to the interface leaks throughout the encoder and no component is guaranteed to remain valid\. The factored representation therefore admits an option a joint encoder does not: keepϕs\\phi\_\{s\}, adapt onlyϕa\\phi\_\{a\}\.

We instantiate two such rewrites\. Actuator degradation applies independent per\-joint gain scaling and polarity inversion\. Action permutation randomly reassigns joint channel indices\. Together they capture the basic ways an action interface can drift, independent scaling and reordering, and serve as a basic simulation of the transfer problem without the confounding factors of full domain shift\. The transition over true torques and the reward are both unchanged; only the path from policy output to actuation is rewritten\. Source and target therefore share the same task, reward, and body, and differ only in the actuator interface, so any retraining cost reflects what the representation failed to preserve\. For each architecture we freeze the largest state\-dependent component \(ϕs\\phi\_\{s\}for FaStR, the joint encoderφ\\varphifor CTRL\-SR\) and reinitialize everything downstream\. Walker is a parity check on matched pre\-shift returns; Humanoid is the diagnostic\.

##### What is preserved, and at what cost\.

On a CP\-factored MDP, which component to keep across an actuator shift is determined by the representation, not a hyperparameter the practitioner has to choose\. A monolithic encoder gives no such handle: the only options are to retrain everything or to freeze arbitrary subsets, neither of which uses structural information about the shift\. A second concern is whether the frozen component is a worse starting point than learning from scratch, which would limit FaStR freeze to a lower final return than CTRL\-SR scratch\. The Humanoid match in Figure[3](https://arxiv.org/html/2607.13498#S4.F3)excludes this: the parameters that CTRL\-SR retrains from scratch are the same ones FaStR keeps unchanged, reusing them matches CTRL\-SR\-scratch on the target task \(Humanoid\-Walk: 393\.7 vs\. 394\.1 under permutation, 401\.8 vs\. 399\.2 under degradation\)\. The modularity is therefore useful in practice and not only as a structural property\.

## 5Related Work

Modern spectral RL methods learn policy\-agnostic features with self\-supervised objectives that decompose the transition kernel: SPEDER\(Renet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib18)\)uses orthogonality\-regularized spectral contrastive learning, CTRL\-SR\(Zhanget al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib10); Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\)uses ranking\-based perturbed NCE, Diff\-SR\(Shribaket al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib21)\)handles multimodal next states with a diffusion objective, and the Spectral Bellman Method\(Nabatiet al\.,[2026](https://arxiv.org/html/2607.13498#bib.bib23)\)uses a Bellman\-aligned principle\. FaStR is orthogonal to this objective\-level progress: it retains the NCE pipeline but replaces the fused encoder with a CP\-factored one, moving factored structure from the objective into the representation itself\. This also separates FaStR from prior uses of multiplicative coupling through FiLM\(Perezet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib39)\), hypernetworks\(Haet al\.,[2017](https://arxiv.org/html/2607.13498#bib.bib40)\), or temporal InfoNCE \(a mutual\-information contrastive loss\) with split encoders\(Zhenget al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib9)\), which do not factorize the transition kernel itself\. Low\-rank MDPs\(Jinet al\.,[2020](https://arxiv.org/html/2607.13498#bib.bib15); Yang and Wang,[2020](https://arxiv.org/html/2607.13498#bib.bib26); Agarwalet al\.,[2020](https://arxiv.org/html/2607.13498#bib.bib27)\)and related exploration, representation, and matrix\-completion analyses\(Modiet al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib33); Ueharaet al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib28); Duet al\.,[2019](https://arxiv.org/html/2607.13498#bib.bib35); Stojanovicet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib22); Dubailet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib31)\)also impose structure on transitions, but treat the joint state\-action feature as a single function\. Other tensor or factored RL methods place structure on the Q\-table, continuous\-action value function, or multi\-agent Q\-tensor\(Rozadaet al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib19),[2025](https://arxiv.org/html/2607.13498#bib.bib20); Lobelet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib5); Mahajanet al\.,[2021](https://arxiv.org/html/2607.13498#bib.bib44)\)\. FaStR sits between these lines by placing factored structure on the transition kernel within a deep spectral representation\-learning framework\.

## 6Discussion and Conclusion

We introduce FaStR, a spectral representation learning framework that decomposes the transition kernel as a three\-way tensor with state, action, and next\-state factors, estimated through a trilinear structured contrastive objective\. The resulting factored class has a quantifiable statistical advantage: transition samples constrain reusable state and action factors rather than isolated state\-action pairs, so the frozen feature can become useful for Bellman backups earlier in training when the CP structure matches the dynamics\. The same separation also supports modular adaptation: when the action\-to\-torque mapping shifts, the state factor transfers intact while only the action factor needs to be retrained\. The broader principle is that representations should structurally mirror the physical factorization of future embodied systems into sensors, actuators, and the dynamics connecting them\.

## References

- FLAMBE: structural complexity and representation learning of low rank MDPs\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.13498#S1.p1.1),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- P\. L\. Bartlett and S\. Mendelson \(2002\)Rademacher and Gaussian complexities: risk bounds and structural results\.Journal of Machine Learning Research3,pp\. 463–482\.Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.3.p1.28)\.
- J\. D\. Carroll and J\. Chang \(1970\)Analysis of individual differences in multidimensional scaling via an N\-way generalization of “Eckart\-Young” decomposition\.Psychometrika35\(3\),pp\. 283–319\.External Links:[Document](https://dx.doi.org/10.1007/BF02310791)Cited by:[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px1.p2.4)\.
- S\. S\. Du, S\. M\. Kakade, J\. D\. Lee, S\. Lovett, G\. Mahajan, W\. Sun, and R\. Wang \(2021\)Bilinear classes: a structural framework for provable generalization in RL\.InProceedings of the International Conference on Machine Learning,Cited by:[§D\.1\.2](https://arxiv.org/html/2607.13498#A4.SS1.SSS2.Px5.p1.3),[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px4.p1.5)\.
- S\. S\. Du, A\. Krishnamurthy, N\. Jiang, A\. Agarwal, M\. Dudík, and J\. Langford \(2019\)Provably efficient RL with rich observations via latent state decoding\.InProceedings of the International Conference on Machine Learning,Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- B\. Dubail, S\. Stojanovic, and A\. Proutiere \(2025\)Shift before you learn: enabling low\-rank representations in reinforcement learning\.arXiv preprint arXiv:2509\.05193\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- R\. M\. Dudley \(1967\)The sizes of compact subsets of Hilbert space and continuity of Gaussian processes\.Journal of Functional Analysis1\(3\),pp\. 290–330\.External Links:[Document](https://dx.doi.org/10.1016/0022-1236%2867%2990017-1)Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.3.p1.17)\.
- S\. Fujimoto, W\. Chang, E\. J\. Smith, S\. S\. Gu, D\. Precup, and D\. Meger \(2023\)For SALE: state\-action representation learning for deep reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.6](https://arxiv.org/html/2607.13498#A2.SS6.p1.20),[Appendix B](https://arxiv.org/html/2607.13498#A2.p1.1),[§4\.1](https://arxiv.org/html/2607.13498#S4.SS1.p1.3)\.
- S\. Fujimoto, H\. van Hoof, and D\. Meger \(2018\)Addressing function approximation error in actor\-critic methods\.InProceedings of the International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.13498#S1.p1.1)\.
- C\. Gao, H\. Sun, N\. Li, D\. Schuurmans, and B\. Dai \(2025\)Spectral representation\-based reinforcement learning\.arXiv preprint arXiv:2512\.15036\.Cited by:[§B\.1](https://arxiv.org/html/2607.13498#A2.SS1.p1.10),[§B\.2](https://arxiv.org/html/2607.13498#A2.SS2.SSS0.Px1.p1.13),[§B\.2](https://arxiv.org/html/2607.13498#A2.SS2.SSS0.Px3.p1.6),[§B\.3](https://arxiv.org/html/2607.13498#A2.SS3.p1.12),[Appendix B](https://arxiv.org/html/2607.13498#A2.p1.1),[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.p1.3),[§D\.1\.1](https://arxiv.org/html/2607.13498#A4.SS1.SSS1.Px1.p1.5),[§1](https://arxiv.org/html/2607.13498#S1.p1.1),[§1](https://arxiv.org/html/2607.13498#S1.p2.1),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p1.5),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p2.1),[§4\.1](https://arxiv.org/html/2607.13498#S4.SS1.p1.3),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- M\. Gutmann and A\. Hyvärinen \(2010\)Noise\-contrastive estimation: a new estimation principle for unnormalized statistical models\.InProceedings of the International Conference on Artificial Intelligence and Statistics,Y\. W\. Teh and M\. Titterington \(Eds\.\),Proceedings of Machine Learning Research, Vol\.9,pp\. 297–304\.Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.p1.3),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p1.5)\.
- D\. Ha, A\. Dai, and Q\. V\. Le \(2017\)HyperNetworks\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the International Conference on Machine Learning,Cited by:[§B\.5](https://arxiv.org/html/2607.13498#A2.SS5.p1.10),[Appendix B](https://arxiv.org/html/2607.13498#A2.p1.1),[§1](https://arxiv.org/html/2607.13498#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.13498#S4.SS1.p1.3)\.
- R\. A\. Harshman \(1970\)Foundations of the PARAFAC procedure: models and conditions for an “explanatory” multi\-modal factor analysis\.UCLA Working Papers in PhoneticsTechnical Report16,University of California, Los Angeles\.Cited by:[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px1.p2.4)\.
- C\. Jin, Z\. Yang, Z\. Wang, and M\. I\. Jordan \(2020\)Provably efficient reinforcement learning with linear function approximation\.InProceedings of the Conference on Learning Theory,Cited by:[§C\.2](https://arxiv.org/html/2607.13498#A3.SS2.SSS0.Px3.p1.11),[§1](https://arxiv.org/html/2607.13498#S1.p1.1),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px1.p1.10),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px1.p1.3),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px3.p1.7),[§3\.2](https://arxiv.org/html/2607.13498#S3.SS2.SSS0.Px2.p1.12),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- T\. G\. Kolda and B\. W\. Bader \(2009\)Tensor decompositions and applications\.SIAM Review51\(3\),pp\. 455–500\.External Links:[Document](https://dx.doi.org/10.1137/07070111X)Cited by:[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px1.p2.4)\.
- H\. Lin, W\. Ding, J\. Chen, L\. Shi, J\. Zhu, B\. Li, and D\. Zhao \(2024\)BECAUSE: bilinear causal representation for generalizable offline model\-based reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px4.p1.5)\.
- S\. Lobel, S\. Rammohan, B\. He, S\. Yu, and G\. Konidaris \(2023\)Q\-functionals for value\-based continuous control\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 8932–8939\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- Z\. Ma and M\. Collins \(2018\)Noise contrastive estimation and negative sampling for conditional models: consistency and statistical efficiency\.arXiv preprint arXiv:1809\.01812\.Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.SSS0.Px2.p1.1),[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.p1.3),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p2.8)\.
- A\. Mahajan, M\. Samvelyan, L\. Mao, V\. Makoviychuk, A\. Garg, J\. Kossaifi, S\. Whiteson, Y\. Zhu, and A\. Anandkumar \(2021\)Tesseract: tensorised actors for multi\-agent reinforcement learning\.InProceedings of the International Conference on Machine Learning,Vol\.139,pp\. 7301–7312\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- C\. McDiarmid \(1989\)On the method of bounded differences\.InSurveys in Combinatorics, 1989,J\. Siemons \(Ed\.\),London Mathematical Society Lecture Note Series, Vol\.141,pp\. 148–188\.Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.3.p1.25)\.
- A\. Modi, J\. Chen, A\. Krishnamurthy, N\. Jiang, and A\. Agarwal \(2024\)Model\-free representation learning and exploration in low\-rank MDPs\.Journal of Machine Learning Research\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- O\. Nabati, B\. Dai, S\. Mannor, and G\. Tennenholtz \(2026\)Spectral Bellman method: unifying representation and exploration in RL\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.13498#S1.p2.1),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- E\. Perez, F\. Strub, H\. De Vries, V\. Dumoulin, and A\. Courville \(2018\)FiLM: visual reasoning with a general conditioning layer\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- A\. Rahimi and B\. Recht \(2007\)Random features for large\-scale kernel machines\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.2](https://arxiv.org/html/2607.13498#A2.SS2.SSS0.Px1.p1.13)\.
- T\. Ren, T\. Zhang, L\. Lee, J\. E\. Gonzalez, D\. Schuurmans, and B\. Dai \(2023\)Spectral decomposition representation for reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.p1.3),[§D\.1\.2](https://arxiv.org/html/2607.13498#A4.SS1.SSS2.Px4.p2.7),[§1](https://arxiv.org/html/2607.13498#S1.p2.1),[§3\.2](https://arxiv.org/html/2607.13498#S3.SS2.SSS0.Px3.p1.2),[§3\.2](https://arxiv.org/html/2607.13498#S3.SS2.SSS0.Px3.p1.3),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- S\. Rozada, S\. Paternain, and A\. G\. Marques \(2024\)Tensor and matrix low\-rank value\-function approximation in reinforcement learning\.IEEE Transactions on Signal Processing72,pp\. 1634–1649\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- S\. Rozada, H\. Wai, and A\. G\. Marques \(2025\)Multilinear tensor low\-rank approximation for policy\-gradient methods in reinforcement learning\.IEEE Transactions on Signal Processing73,pp\. 4906–4920\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- D\. Shribak, C\. Gao, Y\. Li, C\. Xiao, and B\. Dai \(2024\)Diffusion spectral representation for reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.4](https://arxiv.org/html/2607.13498#A2.SS4.p1.21),[§B\.4](https://arxiv.org/html/2607.13498#A2.SS4.p2.13),[Appendix B](https://arxiv.org/html/2607.13498#A2.p1.1),[§D\.1\.2](https://arxiv.org/html/2607.13498#A4.SS1.SSS2.Px4.p2.7),[§1](https://arxiv.org/html/2607.13498#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.13498#S4.SS1.p1.3),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole \(2021\)Score\-based generative modeling through stochastic differential equations\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p2.8)\.
- S\. Stojanovic, Y\. Jedra, and A\. Proutiere \(2023\)Spectral entry\-wise matrix estimation for low\-rank reinforcement learning\.arXiv preprint arXiv:2310\.06793\.Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq, T\. Lillicrap, and M\. Riedmiller \(2018\)DeepMind Control Suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§B\.1](https://arxiv.org/html/2607.13498#A2.SS1.p1.10),[item 3](https://arxiv.org/html/2607.13498#S1.I1.i3.p1.1),[§4\.1](https://arxiv.org/html/2607.13498#S4.SS1.p1.3)\.
- L\. R\. Tucker \(1966\)Some mathematical notes on three\-mode factor analysis\.Psychometrika31\(3\),pp\. 279–311\.External Links:[Document](https://dx.doi.org/10.1007/BF02289464)Cited by:[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px1.p2.4)\.
- M\. Uehara, X\. Zhang, and W\. Sun \(2022\)Representation learning for online and offline RL in low\-rank MDPs\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px1.p1.10),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- A\. van den Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p1.5),[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px3.p1.3)\.
- M\. J\. Wainwright \(2019\)High\-dimensional statistics: a non\-asymptotic viewpoint\.Cambridge University Press\.Cited by:[§C\.4](https://arxiv.org/html/2607.13498#A3.SS4.p1.7),[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.3.p1.17),[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.3.p1.28)\.
- L\. Yang and M\. Wang \(2020\)Reinforcement learning in feature space: matrix bandit, kernels, and regret bound\.InProceedings of the International Conference on Machine Learning,Cited by:[§D\.1\.2](https://arxiv.org/html/2607.13498#A4.SS1.SSS2.Px5.p1.3),[§1](https://arxiv.org/html/2607.13498#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.13498#S3.SS1.SSS0.Px4.p1.5),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- T\. Zhang, T\. Ren, M\. Yang, J\. E\. Gonzalez, D\. Schuurmans, and B\. Dai \(2022\)Making linear MDPs practical via contrastive representation learning\.InProceedings of the International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 26447–26466\.Cited by:[§C\.6](https://arxiv.org/html/2607.13498#A3.SS6.p1.3),[§1](https://arxiv.org/html/2607.13498#S1.p2.1),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p1.5),[§2](https://arxiv.org/html/2607.13498#S2.SS0.SSS0.Px2.p2.1),[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.
- R\. Zheng, X\. Wang, Y\. Sun, S\. Ma, J\. Zhao, H\. Xu, H\. Daumé III, and F\. Huang \(2023\)TACO: temporal latent action\-driven contrastive loss for visual reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2607.13498#S5.p1.1)\.

## Appendix ALimitations\.

The CP formϕs​\(s\)⊙ϕa​\(a\)\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)treats the state and the action as two atomic axes\. Environments with strongly coupled joints and tight state–action interdependence, such as Ant\-style locomotion, fall outside the regime this two\-way split captures cleanly, since the encoder is forced to absorb all sub\-axis interactions internally\. A hierarchical factorization over actuator or body\-part groups, e\.g\.ϕs​\(s\)=ϕtorso⊙ϕlimbs\\phi\_\{s\}\(s\)=\\phi\_\{\\text\{torso\}\}\\odot\\phi\_\{\\text\{limbs\}\}or a tree\-shaped CP, fits inside the same framework and is a clean next step\.Second, DiagonalMMcorresponds to CP rank one along the next\-state axis\. A low\-rank Tucker coreG∈ℝd×dG\\in\\mathbb\{R\}^\{d\\times d\}recovers more interaction terms but enlarges the effective NCE dimension fromddtod2d^\{2\}, reintroducing the constraint that motivated the diagonal choice\. Mapping the rank–capacity trade\-off, e\.g\. via low\-rank rather than diagonalMM, would close the gap between the CP and Tucker ends of this spectrum\. Also, all experiments are conducted in the DMControl proprioceptive locomotion suite\. Pixel\-input variants and physical\-robot deployment are not evaluated; the latter introduces sensor noise, latency, and actuator drift that the actuator\-shift experiments only partially approximate\.

## Appendix BImplementation Details

This appendix specifies the full training and evaluation protocol used in Section[4](https://arxiv.org/html/2607.13498#S4), together with the architectural and optimization hyperparameters of FaStR and the five baselines reported in Table[1](https://arxiv.org/html/2607.13498#S4.T1): CTRL\-SR\[Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\], Diff\-SR\[Shribaket al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib21)\], SAC\[Haarnojaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib14)\]in two width\-depth variants \(3\-layer and 2\-layer\), and TD7\[Fujimotoet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib41)\]\. Hyperparameters that are common across methods \(environment configuration, replay, evaluation cadence, total interaction budget\) are listed once in Section[B\.1](https://arxiv.org/html/2607.13498#A2.SS1); method\-specific settings are listed in Sections[B\.2](https://arxiv.org/html/2607.13498#A2.SS2)–[B\.6](https://arxiv.org/html/2607.13498#A2.SS6)and summarized side\-by\-side in Table[2](https://arxiv.org/html/2607.13498#A2.T2)\. We follow the convention of reporting every hyperparameter that was set in our runs, including those left at the original authors’ defaults, so that the protocol is self\-contained\.

### B\.1Shared Training Protocol

All experiments use the DeepMind Control Suite\[Tassaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib11)\]with proprioceptive observations, action repeat \(frame skip\) of 2, and episode length 1000 environment steps\. We train each agent for1×1061\{\\times\}10^\{6\}environment frames \(i\.e\.5×1055\{\\times\}10^\{5\}agent steps after action repeat\) per task\. Each agent maintains a single uniform\-sampling replay buffer of capacity1×1061\{\\times\}10^\{6\}transitions, with the first10,00010\{,\}000frames collected by a uniform random policy as a warm start before any gradient update\. Evaluation is performed every10,00010\{,\}000environment frames by rolling out the deterministic actor \(for TD3\-based methods: TD3 mean action; for SAC: tanh\-Gaussian mean\) for1010episodes; we report the mean undiscounted return over the last 10 evaluation checkpoints, averaged over55random seeds, with shaded bands or±1\\pm 1standard deviations across seeds\. Observation normalization \(running mean and standard deviation\) is enabled for the spectral methods \(FaStR and CTRL\-SR\), followingGaoet al\.\[[2025](https://arxiv.org/html/2607.13498#bib.bib12)\], and disabled for the SAC and TD7 baselines, matching their original implementations\. All methods use the discount factorγ=0\.99\\gamma\{=\}0\.99, the same set of seeds\{0,1,2,3,42\}\\\{0,1,2,3,42\\\}, and a single update per environment step \(update\-to\-data ratio 1\)\.

### B\.2FaStR \(Ours\)

##### Architecture\.

All encoders, the noise schedule, and design defaults that are not explicitly varied below followGaoet al\.\[[2025](https://arxiv.org/html/2607.13498#bib.bib12)\]\. Encoders are ResidualMLPs with LayerNorm, ELU activations, and Mish activations inside the contrastive head\. The state factorϕs\\phi\_\{s\}mapsdim\(𝒮\)→512→512→d\\dim\(\\mathcal\{S\}\)\\to 512\\to 512\\to d; the action factorϕa\\phi\_\{a\}mapsdim\(𝒜\)→512→512→d\\dim\(\\mathcal\{A\}\)\\to 512\\to 512\\to d; the kernel encodermmmapsdim\(𝒮\)\+128→512→512→d\\dim\(\\mathcal\{S\}\)\+128\\to 512\\to 512\\to d, where the additional 128 dimensions encode the perturbation level via a sinusoidal time embedding\. We setd=512d\{=\}512throughout\. The factored featureψ​\(s,a\)=ϕs​\(s\)⊙ϕa​\(a\)\\psi\(s,a\)=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)is fed to a twin Q critic consisting of a random Fourier features \(RFF\)\[Rahimi and Recht,[2007](https://arxiv.org/html/2607.13498#bib.bib17)\]projection of width10241024followed by a 2\-layer MLP head\[512\]\[512\]with LayerNorm and ELU; this critic is architecturally identical to the CTRL\-SR critic so that the only varying component is the encoder\. The actor is a deterministic policy network with hidden sizes\[512,512,512\]\[512,512,512\], ELU activations, and a finaltanh\\tanhthat maps to\[−1,1\]dim\(𝒜\)\[\-1,1\]^\{\\dim\(\\mathcal\{A\}\)\}\.

##### Optimization\.

All four networks \(state factor, action factor, kernel encoder, critic\) are trained with Adam \(default\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\), no weight decay\)\. Critic and actor learning rates are3×10−43\{\\times\}10^\{\-4\}; the representation learning rate \(jointly applied toϕs,ϕa,m\\phi\_\{s\},\\phi\_\{a\},m\) is1×10−41\{\\times\}10^\{\-4\}\. Polyak averaging \(i\.e\.,θ¯←\(1−τ\)​θ¯\+τ​θ\\bar\{\\theta\}\\leftarrow\(1\-\\tau\)\\bar\{\\theta\}\+\\tau\\theta\) withτ=0\.005\\tau\{=\}0\.005is used for both the critic and the representation targets; actor and critic updates are performed every environment step \(actor\_update\_freq=1\)\. The mini\-batch size isN=512N\{=\}512, matching the negative\-sample countK=N=512K\{=\}N\{=\}512implicit in the in\-batch RP\-NCE objective\. Exploration uses Gaussian action noise𝒩​\(0,σ=0\.2\)\\mathcal\{N\}\(0,\\sigma\{=\}0\.2\)clipped to\[−1,1\]dim\(𝒜\)\[\-1,1\]^\{\\dim\(\\mathcal\{A\}\)\}; target action smoothing uses𝒩​\(0,σ~=0\.2\)\\mathcal\{N\}\(0,\\tilde\{\\sigma\}\{=\}0\.2\)clipped to\[−0\.3,0\.3\]\[\-0\.3,0\.3\]\. Gradients of the critic loss are not back\-propagated through the representation \(back\_critic\_grad=false\)\. One training step is summarized in Algorithm[1](https://arxiv.org/html/2607.13498#alg1)\.

##### Representation objective\.

We use the ranking\-based perturbed NCE loss \([7](https://arxiv.org/html/2607.13498#S3.E7)\) to train the trilinear transition\-ratio score, withM=25M\{=\}25noise levels on a variance\-preserving \(VP\) schedule\. The ranking variant is enabled \(ranking=true\), and the auxiliary bilinear\-MDP regression term ofGaoet al\.\[[2025](https://arxiv.org/html/2607.13498#bib.bib12)\]is kept at its default weightβ=1\.0\\beta\{=\}1\.0\. The combined training loss isℓFaStR\+λr​ℓreward\+λQ​ℓcritic\\ell\_\{\\mathrm\{FaStR\}\}\+\\lambda\_\{r\}\\,\\ell\_\{\\mathrm\{reward\}\}\+\\lambda\_\{Q\}\\,\\ell\_\{\\mathrm\{critic\}\}withλctrl=λr=λQ=1\.0\\lambda\_\{\\mathrm\{ctrl\}\}\{=\}\\lambda\_\{r\}\{=\}\\lambda\_\{Q\}\{=\}1\.0; the reward head is a 2\-layer MLP\[512\]\[512\]on top ofψ\\psi, identical in structure to the CTRL\-SR reward head\.

Algorithm 1FaStR: one training step\.sg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\): stop\-gradient\.0:Encoders

ϕs\\phi\_\{s\},

ϕa\\phi\_\{a\},

mm; critic

QθQ\_\{\\theta\}; actor

πξ\\pi\_\{\\xi\}; target copies

ϕ¯s\\bar\{\\phi\}\_\{s\},

ϕ¯a\\bar\{\\phi\}\_\{a\},

m¯\\bar\{m\},

Q¯\\bar\{Q\}
1:Sample mini\-batch

\{\(si,ai,ri,si′\)\}i=1N\\\{\(s\_\{i\},a\_\{i\},r\_\{i\},s^\{\\prime\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}; draw

K−1K\{\-\}1negatives per transition

2:Representation:

ψi←ϕs​\(si\)⊙ϕa​\(ai\)\\psi\_\{i\}\\leftarrow\\phi\_\{s\}\(s\_\{i\}\)\\odot\\phi\_\{a\}\(a\_\{i\}\)for all

ii
3:Update

\(ϕs,ϕa,m\)\(\\phi\_\{s\},\\phi\_\{a\},m\)by minimizing

ℓFaStR\\ell\_\{\\text\{FaStR\}\}\(Eq\.[7](https://arxiv.org/html/2607.13498#S3.E7)\)

4:Critic:

ψ¯i′←sg⁡\(ϕ¯s​\(si′\)\)⊙sg⁡\(ϕ¯a​\(πξ​\(si′\)\)\)\\bar\{\\psi\}^\{\\prime\}\_\{i\}\\leftarrow\\operatorname\{sg\}\(\\bar\{\\phi\}\_\{s\}\(s^\{\\prime\}\_\{i\}\)\)\\odot\\operatorname\{sg\}\(\\bar\{\\phi\}\_\{a\}\(\\pi\_\{\\xi\}\(s^\{\\prime\}\_\{i\}\)\)\);

yi←ri\+γ​Q¯​\(ψ¯i′\)y\_\{i\}\\leftarrow r\_\{i\}\+\\gamma\\,\\bar\{Q\}\(\\bar\{\\psi\}^\{\\prime\}\_\{i\}\)
5:Update

θ\\thetaby minimizing

1N​∑i\(Qθ​\(ψi\)−yi\)2\\frac\{1\}\{N\}\\sum\_\{i\}\(Q\_\{\\theta\}\(\\psi\_\{i\}\)\-y\_\{i\}\)^\{2\}
6:Actor:Update

ξ\\xiby maximizing

1N​∑iQθ​\(sg⁡\(ϕs​\(si\)\)⊙sg⁡\(ϕa​\(πξ​\(si\)\)\)\)\\frac\{1\}\{N\}\\sum\_\{i\}Q\_\{\\theta\}\\\!\\bigl\(\\operatorname\{sg\}\(\\phi\_\{s\}\(s\_\{i\}\)\)\\odot\\operatorname\{sg\}\(\\phi\_\{a\}\(\\pi\_\{\\xi\}\(s\_\{i\}\)\)\)\\bigr\)
7:Targets:

\(ϕ¯s,ϕ¯a,m¯,Q¯\)←τ​\(ϕs,ϕa,m,Qθ\)\+\(1−τ\)​\(ϕ¯s,ϕ¯a,m¯,Q¯\)\(\\bar\{\\phi\}\_\{s\},\\bar\{\\phi\}\_\{a\},\\bar\{m\},\\bar\{Q\}\)\\leftarrow\\tau\\,\(\\phi\_\{s\},\\phi\_\{a\},m,Q\_\{\\theta\}\)\+\(1\{\-\}\\tau\)\\,\(\\bar\{\\phi\}\_\{s\},\\bar\{\\phi\}\_\{a\},\\bar\{m\},\\bar\{Q\}\)

### B\.3CTRL\-SR

CTRL\-SR\[Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\]serves as the most controlled baseline: the only architectural difference from FaStR is that the transition\-ratio logit uses a single monolithic encoderφ:𝒮×𝒜→ℝd\\varphi:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\}taking the concatenation\[s;a\]\[s;a\]as input, rather than a CP trilinear score built from separate state and action factors\. The encoderφ\\varphiis a 2\-layer ResidualMLP\[dim\(𝒮\)\+dim\(𝒜\)\]→512→512→d\[\\dim\(\\mathcal\{S\}\)\+\\dim\(\\mathcal\{A\}\)\]\\to 512\\to 512\\to dwithd=512d\{=\}512, LayerNorm, and ELU activations; the kernel encodermm, RFF\-MLP critic, deterministic actor\[512,512,512\]\[512,512,512\], reward head, optimizer \(Adam\), learning rates \(3×10−43\{\\times\}10^\{\-4\}for actor/critic and1×10−41\{\\times\}10^\{\-4\}for representation\), Polyak factorτ=0\.005\\tau\{=\}0\.005, target/exploration action noise, batch sizeN=512N\{=\}512, RP\-NCE noise levelsM=25M\{=\}25, ranking objective, and update cadence are all identical to those of FaStR\. We use the authors’ open\-source implementation without modification\.

### B\.4Diff\-SR

Diff\-SR\[Shribaket al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib21)\]is the second spectral RL baseline\. It uses the same TD3 actor and critic backbone as CTRL\-SR and FaStR, but replaces the ranking\-based perturbed NCE objective with a denoising\-diffusion next\-state model: the joint encoderϕ:𝒮×𝒜→ℝd\\phi:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\}and the kernel encoderμ:𝒮×\[1,T\]→ℝd×dim\(𝒮\)\\mu:\\mathcal\{S\}\\times\[1,T\]\\to\\mathbb\{R\}^\{d\\times\\dim\(\\mathcal\{S\}\)\}are trained by score matching against the variance\-preserving \(VP\) noise schedule ofShribaket al\.\[[2024](https://arxiv.org/html/2607.13498#bib.bib21)\], with the predicted noise factorized asϵ^​\(s,a,st′,t\)=ϕ​\(s,a\)⊤​μ​\(st′,t\)\\hat\{\\epsilon\}\(s,a,s^\{\\prime\}\_\{t\},t\)=\\phi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\_\{t\},t\)\. The state and action inputs toϕ\\phiare first projected by 2\-layer Mish MLPs of widths\[256,128\]\[256,128\]to a shared embedding dimension of128128each, concatenated to a256256\-dim vector, and passed to a 3\-layer ResidualMLP256→512→512→512→d256\\to 512\\to 512\\to 512\\to dwithd=512d\{=\}512\. The kernel encoderμ\\mutakes\(st′,t\)\(s^\{\\prime\}\_\{t\},t\)as input, wherettis encoded by a sinusoidal positional embedding of width128128followed by a 2\-layer Mish MLP, and outputs ad×dim\(𝒮\)d\\times\\dim\(\\mathcal\{S\}\)matrix via a 3\-layer ResidualMLP\[dim\(𝒮\)\+128\]→512→512→512→d⋅dim\(𝒮\)\[\\dim\(\\mathcal\{S\}\)\+128\]\\to 512\\to 512\\to 512\\to d\\cdot\\dim\(\\mathcal\{S\}\)\. The diffusion usesT=50T\{=\}50sample steps with the noisy state clamped to\[−10,10\]\[\-10,10\]\. The critic is the same RFF\-MLP twin Q as CTRL\-SR \(RFF projection of width10241024followed by a 1\-hidden\-layer\[512\]\[512\]head\); the deterministic actor is a\[512,512,512\]\[512,512,512\]MLP with LayerNorm and a finaltanh\\tanh; the reward head is an RFF\-MLP with the same shape as the critic head\.

All four networks are trained with Adam\. Critic and actor learning rates are3×10−43\{\\times\}10^\{\-4\}; the representation learning rate \(jointly applied toϕ,μ\\phi,\\mu, and the reward head\) is1×10−41\{\\times\}10^\{\-4\}\. Polyak averaging usesτ=0\.005\\tau\{=\}0\.005, and actor and critic updates are performed every environment step \(actor\_update\_freq=1,feature\_update\_ratio=1\)\. Mini\-batch size isN=512N\{=\}512, matching the spectral baselines\. Exploration uses Gaussian noise𝒩​\(0,σ=0\.2\)\\mathcal\{N\}\(0,\\sigma\{=\}0\.2\)clipped to\[−1,1\]dim\(𝒜\)\[\-1,1\]^\{\\dim\(\\mathcal\{A\}\)\}; target action smoothing uses𝒩​\(0,σ~=0\.2\)\\mathcal\{N\}\(0,\\tilde\{\\sigma\}\{=\}0\.2\)clipped to\[−0\.3,0\.3\]\[\-0\.3,0\.3\]\. The combined training loss isλdiff​ℓdiffusion\+λr​ℓreward\+λQ​ℓcritic\\lambda\_\{\\mathrm\{diff\}\}\\,\\ell\_\{\\mathrm\{diffusion\}\}\+\\lambda\_\{r\}\\,\\ell\_\{\\mathrm\{reward\}\}\+\\lambda\_\{Q\}\\,\\ell\_\{\\mathrm\{critic\}\}withλdiff=1\.0\\lambda\_\{\\mathrm\{diff\}\}\{=\}1\.0,λr=0\.1\\lambda\_\{r\}\{=\}0\.1, andλQ=1\.0\\lambda\_\{Q\}\{=\}1\.0; gradients of the critic loss are not back\-propagated through the representation \(back\_critic\_grad=false\)\. Observation normalization is enabled, matching CTRL\-SR and FaStR\. We use the open\-source implementation accompanyingShribaket al\.\[[2024](https://arxiv.org/html/2607.13498#bib.bib21)\]without modification\.

### B\.5SAC

SAC\[Haarnojaet al\.,[2018](https://arxiv.org/html/2607.13498#bib.bib14)\]uses a stochastic tanh\-Gaussian policy and a twin Q critic with ELU activations and LayerNorm\. The critic takes the raw\(s,a\)\(s,a\)concatenation as input; no representation learning is performed\. We report two width\-depth variants\. SAC\(3\-layer\) uses hidden sizes\[1024,1024,1024\]\[1024,1024,1024\]for both the actor and the critic, matching the standard SAC configuration on DM Control Suite\. SAC\(2\-layer\) uses hidden sizes\[1024,1024\]\[1024,1024\]for both, isolating the effect of removing one hidden layer at the same per\-layer width\. All other settings are shared between the two variants\. FollowingHaarnojaet al\.\[[2018](https://arxiv.org/html/2607.13498#bib.bib14)\], the entropy coefficientα\\alphais learned online \(automatic entropy tuning\) toward a target entropy of−dim\(𝒜\)\-\\dim\(\\mathcal\{A\}\), optimized via Adam and learning rate3×10−43\{\\times\}10^\{\-4\}from an initial valueα0=0\.2\\alpha\_\{0\}\{=\}0\.2\. Actor, critic, and entropy coefficient are all trained with Adam at learning rate3×10−43\{\\times\}10^\{\-4\}; soft target updates use Polyak factorτ=0\.005\\tau\{=\}0\.005at every environment step, and the actor is updated every step \(actor\_update\_freq=1\)\. Mini\-batch size is512512for parity with the spectral methods\. There is no exploration noise injected on top of the policy stochasticity, no observation normalization, and no gradient clipping\.

### B\.6TD7

TD7\[Fujimotoet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib41)\]extends TD3 with a learned state\-action encoder \(z​s,z​s​azs,zsa\) trained by latent dynamics regression, Loss\-Adjusted Prioritized \(LAP\) replay with per\-transition priorities, behavior\-cloning regularization at the policy update, and policy checkpointing\. We use the official implementation released by the authors with the original DM Control Suite hyperparameters; the only configuration we set is the task identifier and the seed\. For completeness: encoder, actor, and critic each use 2\-hidden\-layer MLPs of width256256with ELU activations \(ReLU for the actor\) and the AvgL1Norm normalization ofFujimotoet al\.\[[2023](https://arxiv.org/html/2607.13498#bib.bib41)\]\(per\-layer activation rescaling\); the encoded state\-action dimension iszdim=256z\_\{\\mathrm\{dim\}\}\{=\}256\. Optimizer is Adam with learning rate3×10−43\{\\times\}10^\{\-4\}for all three networks\. Replay uses LAP withα=0\.4\\alpha\{=\}0\.4,min\\min\-priority11, and Huber critic loss; the discount isγ=0\.99\\gamma\{=\}0\.99and the target network is updated by a hard copy every250250updates \(rather than Polyak averaging\)\. Exploration uses Gaussian noise𝒩​\(0,0\.1\)\\mathcal\{N\}\(0,0\.1\)clipped to\[−1,1\]\[\-1,1\]; target policy smoothing uses𝒩​\(0,0\.2\)\\mathcal\{N\}\(0,0\.2\)clipped to\[−0\.5,0\.5\]\[\-0\.5,0\.5\]; the actor is updated every22critic updates with behavior\-cloning weightλBC=0\.1\\lambda\_\{\\mathrm\{BC\}\}\{=\}0\.1\. Policy checkpointing uses up to2020episodes per checkpoint and starts after7\.5×1057\.5\{\\times\}10^\{5\}agent steps with weight\-reset coefficient0\.90\.9\. Mini\-batch size is256256\(the value used in the original TD7 paper\); we did not tune this for parity with the spectral methods\. Random exploration runs for the first25,00025\{,\}000agent steps before any training begins\.

### B\.7Hyperparameter Summary

Table[2](https://arxiv.org/html/2607.13498#A2.T2)consolidates the values reported above\. Dashes \(—\) indicate that a hyperparameter does not apply to a given method \(e\.g\., a representation learning rate for SAC\)\. Where a method has additional method\-specific hyperparameters not listed in the table \(LAPα\\alphafor TD7, BC weight, target entropy for SAC, RP\-NCE noise schedule for the spectral methods\), we have specified them in the corresponding paragraph above and use the original authors’ default values\.

Table 2:Hyperparameters for FaStR and the five baselines used in Table[1](https://arxiv.org/html/2607.13498#S4.T1)\. Values shared across all methods \(discountγ=0\.99\\gamma\{=\}0\.99,1×1061\{\\times\}10^\{6\}environment frames, replay capacity1×1061\{\\times\}10^\{6\}, action repeat22,55seeds,1010evaluation episodes every10,00010\{,\}000frames\) are described in Section[B\.1](https://arxiv.org/html/2607.13498#A2.SS1)\.

## Appendix CProof Details

### C\.1Notation, Conventions, and Standing Assumptions

We work with a discounted MDP\(𝒮,𝒜,P,r,γ,ρ0\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma,\\rho\_\{0\}\)on a continuous state space𝒮\\mathcal\{S\}and continuous action space𝒜\\mathcal\{A\}, withγ∈\[0,1\)\\gamma\\in\[0,1\)and reward bounded byRmaxR\_\{\\max\}\. We assume the existence of a fixed reference*probability measure*ν\\nuon𝒮\\mathcal\{S\}that dominates every transition kernel, so thatP\(⋅∣s,a\)≪νP\(\\cdot\\mid s,a\)\\ll\\nuand admits a densityP​\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)with respect toν\\nu\. All integrals over𝒮\\mathcal\{S\}are then taken with respect toν\\nu, i\.e\.∫𝒮f​\(s′\)​𝑑s′≡∫𝒮f​\(s′\)​𝑑ν​\(s′\)\\int\_\{\\mathcal\{S\}\}f\(s^\{\\prime\}\)\\,ds^\{\\prime\}\\equiv\\int\_\{\\mathcal\{S\}\}f\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\. This is the standard setting forL2​\(ν\)L^\{2\}\(\\nu\)\-based representation\-learning analyses and fixes the measure with respect to which all densities and risks are evaluated\.

##### Symbols\.

- •ϕs:𝒮→ℝd\\phi\_\{s\}:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\},ϕa:𝒜→ℝd\\phi\_\{a\}:\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\},m:𝒮→ℝdm:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\}: state encoder, action encoder, kernel encoder\.
- •ψ​\(s,a\):=ϕs​\(s\)⊙ϕa​\(a\)∈ℝd\\psi\(s,a\):=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\\in\\mathbb\{R\}^\{d\}:*factored*\(Hadamard\) feature\.
- •φ​\(s,a\)∈ℝd\\varphi\(s,a\)\\in\\mathbb\{R\}^\{d\},μ:𝒮→ℝd\\mu:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\}:*joint*\(monolithic\) encoder and kernel encoder\.
- •ℱs,ℱa,ℱm,ℱs​a\\mathcal\{F\}\_\{s\},\\mathcal\{F\}\_\{a\},\\mathcal\{F\}\_\{m\},\\mathcal\{F\}\_\{sa\}: hypothesis classes forϕs,ϕa,m,φ\\phi\_\{s\},\\phi\_\{a\},m,\\varphi\.
- •𝒢fac:=\{\(s,a,s′\)↦\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\):ϕs∈ℱs,ϕa∈ℱa,m∈ℱm\}\\mathcal\{G\}\_\{\\mathrm\{fac\}\}:=\\\{\(s,a,s^\{\\prime\}\)\\mapsto\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\):\\phi\_\{s\}\\in\\mathcal\{F\}\_\{s\},\\phi\_\{a\}\\in\\mathcal\{F\}\_\{a\},m\\in\\mathcal\{F\}\_\{m\}\\\}\.
- •𝒢joint:=\{\(s,a,s′\)↦φ​\(s,a\)⊤​μ​\(s′\):φ∈ℱs​a,μ∈ℱm\}\\mathcal\{G\}\_\{\\mathrm\{joint\}\}:=\\\{\(s,a,s^\{\\prime\}\)\\mapsto\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\):\\varphi\\in\\mathcal\{F\}\_\{sa\},\\mu\\in\\mathcal\{F\}\_\{m\}\\\}\.

##### Boundedness assumptions\.

We assume there exist constantsBϕ,Bφ,Bm,∞,Bm,1,Bg≥1B\_\{\\phi\},B\_\{\\varphi\},B\_\{m,\\infty\},B\_\{m,1\},B\_\{g\}\\geq 1such that, uniformly over their hypothesis classes,

supϕs∈ℱssups∈𝒮‖ϕs​\(s\)‖2,supϕa∈ℱasupa∈𝒜‖ϕa​\(a\)‖2\\displaystyle\\sup\_\{\\phi\_\{s\}\\in\\mathcal\{F\}\_\{s\}\}\\sup\_\{s\\in\\mathcal\{S\}\}\\\|\\phi\_\{s\}\(s\)\\\|\_\{2\},\\,\\sup\_\{\\phi\_\{a\}\\in\\mathcal\{F\}\_\{a\}\}\\sup\_\{a\\in\\mathcal\{A\}\}\\\|\\phi\_\{a\}\(a\)\\\|\_\{2\}≤Bϕ,\\displaystyle\\leq B\_\{\\phi\},supφ∈ℱs​asup\(s,a\)‖φ​\(s,a\)‖2\\displaystyle\\sup\_\{\\varphi\\in\\mathcal\{F\}\_\{sa\}\}\\sup\_\{\(s,a\)\}\\\|\\varphi\(s,a\)\\\|\_\{2\}≤Bφ,\\displaystyle\\leq B\_\{\\varphi\},suph∈ℱmsups′∈𝒮‖h​\(s′\)‖2\\displaystyle\\sup\_\{h\\in\\mathcal\{F\}\_\{m\}\}\\sup\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\\|h\(s^\{\\prime\}\)\\\|\_\{2\}≤Bm,∞,\\displaystyle\\leq B\_\{m,\\infty\},suph∈ℱm∫𝒮‖h​\(s′\)‖2​𝑑ν​\(s′\)\\displaystyle\\sup\_\{h\\in\\mathcal\{F\}\_\{m\}\}\\int\_\{\\mathcal\{S\}\}\\\|h\(s^\{\\prime\}\)\\\|\_\{2\}\\,d\\nu\(s^\{\\prime\}\)≤Bm,1,\\displaystyle\\leq B\_\{m,1\},and\|g​\(s,a,s′\)\|≤Bg\|g\(s,a,s^\{\\prime\}\)\|\\leq B\_\{g\}for everyg∈𝒢fac∪𝒢jointg\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\cup\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\. Thus both the factored kernel encodermmand the joint kernel encoderμ\\muobey the same pointwise andL1​\(ν\)L^\{1\}\(\\nu\)bounds\. For a single continuous encoder on compact𝒮\\mathcal\{S\}these bounds are automatic; for a hypothesis class we impose them uniformly\. We separate the pointwise boundBm,∞B\_\{m,\\infty\}\(used for covering numbers and sup\-norm Lipschitz arguments\) from the integral boundBm,1B\_\{m,1\}\(used to verify that integrals againstVπV^\{\\pi\}are finite\); the two regimes appear in different proofs\.

##### Rademacher complexity and covering numbers\.

For a fixed sampleS=\(z1,…,zn\)S=\(z\_\{1\},\\dots,z\_\{n\}\)and a scalar\-valued classℋ\\mathcal\{H\}, define the empirical absolute Rademacher complexity

ℛ^S​\(ℋ\):=𝔼σ​\[suph∈ℋ\|1n​∑i=1nσi​h​\(zi\)\|\],σi∼iidUnif​\{±1\}\.\\widehat\{\\mathcal\{R\}\}\_\{S\}\(\\mathcal\{H\}\):=\\mathbb\{E\}\_\{\\sigma\}\\\!\\left\[\\sup\_\{h\\in\\mathcal\{H\}\}\\left\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sigma\_\{i\}\\,h\(z\_\{i\}\)\\right\|\\right\],\\qquad\\sigma\_\{i\}\\stackrel\{\{\\scriptstyle\\mathrm\{iid\}\}\}\{\{\\sim\}\}\\mathrm\{Unif\}\\\{\\pm 1\\\}\.The expected Rademacher complexity is

ℛn​\(ℋ\):=𝔼S​\[ℛ^S​\(ℋ\)\]\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\):=\\mathbb\{E\}\_\{S\}\\\!\\left\[\\widehat\{\\mathcal\{R\}\}\_\{S\}\(\\mathcal\{H\}\)\\right\]\.All finite\-sample bounds use this expected version\. Set\-inclusion arguments hold pointwise forℛ^S\\widehat\{\\mathcal\{R\}\}\_\{S\}and therefore also after taking expectation overSS\. For scalar\-valued classes,𝒩​\(ℋ,ϵ\)\\mathcal\{N\}\(\\mathcal\{H\},\\epsilon\)denotes the covering number underd∞​\(h,h′\)=supz\|h​\(z\)−h′​\(z\)\|d\_\{\\infty\}\(h,h^\{\\prime\}\)=\\sup\_\{z\}\|h\(z\)\-h^\{\\prime\}\(z\)\|\. For vector\-valued encoder classes such asℱs,ℱa,ℱm,ℱs​a\\mathcal\{F\}\_\{s\},\\mathcal\{F\}\_\{a\},\\mathcal\{F\}\_\{m\},\\mathcal\{F\}\_\{sa\}, the covering metric is

d∞​\(f,f′\)=supx‖f​\(x\)−f′​\(x\)‖2\.d\_\{\\infty\}\(f,f^\{\\prime\}\)=\\sup\_\{x\}\\\|f\(x\)\-f^\{\\prime\}\(x\)\\\|\_\{2\}\.

##### Best\-in\-class CP error\.

For non\-realizable settings we write

ϵCP2:=infg∈𝒢fac𝔼\(s,a\)∼ρ​\[∫𝒮\(P​\(s′∣s,a\)−g​\(s,a,s′\)\)2​𝑑ν​\(s′\)\]\.\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}\\;:=\\;\\inf\_\{g\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}\\;\\mathbb\{E\}\_\{\(s,a\)\\sim\\rho\}\\\!\\left\[\\int\_\{\\mathcal\{S\}\}\\bigl\(P\(s^\{\\prime\}\\mid s,a\)\-g\(s,a,s^\{\\prime\}\)\\bigr\)^\{2\}\\,d\\nu\(s^\{\\prime\}\)\\right\]\.Under CP realizability \(Assumption[1](https://arxiv.org/html/2607.13498#Thmassumption1)\),ϵCP=0\\epsilon\_\{\\mathrm\{CP\}\}=0\.

##### Proof architecture\.

The results in this appendix support a single claim: the CP factorization of the transition kernel gives a representation whose statistical complexity decomposes additively in\(ds,da\)\(d\_\{s\},d\_\{a\}\), whereas the generic uniform\-convergence bound for an unrestricted joint encoder scales with the full joint dimensionds\+dad\_\{s\}\+d\_\{a\}\. Appendix[C\.2](https://arxiv.org/html/2607.13498#A3.SS2)derives the Q\-linearity stated in Section[3\.1](https://arxiv.org/html/2607.13498#S3.SS1): under CP factorization, everyQπQ^\{\\pi\}is linear in the Hadamard feature, so the factored class is matched to the structure of the transition kernel rather than chosen as an architectural convenience\. Appendix[C\.3](https://arxiv.org/html/2607.13498#A3.SS3)proves Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1), the main technical step: the Hadamard product converts the joint covering of𝒢fac\\mathcal\{G\}\_\{\\mathrm\{fac\}\}into a sum of three independent component coverings, and the same set\-inclusion argument gives a Rademacher ordering between the two classes\. Appendix[C\.5](https://arxiv.org/html/2607.13498#A3.SS5)states this Rademacher monotonicity on its own, since it follows from set inclusion without needing the covering bound\. Appendix[C\.6](https://arxiv.org/html/2607.13498#A3.SS6)converts the Rademacher bound into an excess\-risk bound for the spectralL2L\_\{2\}objective with constants made explicit \(Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)\)\. Appendix[C\.7](https://arxiv.org/html/2607.13498#A3.SS7)converts the additive vs\. multiplicative entropy gap into the sufficient\-sample\-size gap implied by these bounds \(Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)\)\. It ends with a same\-nncomparison that holds without any Lipschitz instantiation: under the containment condition, both Rademacher terms in Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)are no larger for the factored class, so the certified upper bound on representation error is no larger at every sample size \(Remark[C\.7](https://arxiv.org/html/2607.13498#A3.SS7.SSS0.Px3)\)\.

### C\.2Derivation of the Q\-linearity under CP

The CP transition structure makes everyQπQ^\{\\pi\}linear in the Hadamard featureψ\\psi\. This appendix gives the linear\-MDP derivation specialized to the CP feature, together with norm bounds on the value coefficients used downstream\. We spell out the interchange of summation and integration and the norm estimates in detail\.

##### Setup\.

Suppose Assumption[1](https://arxiv.org/html/2607.13498#Thmassumption1)holds\. That is, the reward decomposes as

r​\(s,a\)=ψ​\(s,a\)⊤​θr,r\(s,a\)=\\psi\(s,a\)^\{\\top\}\\theta\_\{r\},where

ψ​\(s,a\)=ϕs​\(s\)⊙ϕa​\(a\)∈ℝd,ψk​\(s,a\)=ϕs,k​\(s\)​ϕa,k​\(a\),\\psi\(s,a\)=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\\in\\mathbb\{R\}^\{d\},\\qquad\\psi\_\{k\}\(s,a\)=\\phi\_\{s,k\}\(s\)\\phi\_\{a,k\}\(a\),and the transition density factorizes as

P​\(s′∣s,a\)=∑k=1dϕs,k​\(s\)​ϕa,k​\(a\)​mk​\(s′\)=∑k=1dψk​\(s,a\)​mk​\(s′\)\.P\(s^\{\\prime\}\\mid s,a\)=\\sum\_\{k=1\}^\{d\}\\phi\_\{s,k\}\(s\)\\,\\phi\_\{a,k\}\(a\)\\,m\_\{k\}\(s^\{\\prime\}\)=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)m\_\{k\}\(s^\{\\prime\}\)\.\(9\)Here eachmkm\_\{k\}is measurable and belongs toL1​\(ν\)L^\{1\}\(\\nu\)\. We also use the boundedness conventions of Appendix[C\.1](https://arxiv.org/html/2607.13498#A3.SS1); in particular, writing

m​\(s′\)=\(m1​\(s′\),…,md​\(s′\)\)⊤,m\(s^\{\\prime\}\)=\(m\_\{1\}\(s^\{\\prime\}\),\\ldots,m\_\{d\}\(s^\{\\prime\}\)\)^\{\\top\},we have

∫𝒮‖m​\(s′\)‖2​𝑑ν​\(s′\)≤Bm,1\.\\int\_\{\\mathcal\{S\}\}\\\|m\(s^\{\\prime\}\)\\\|\_\{2\}\\,d\\nu\(s^\{\\prime\}\)\\leq B\_\{m,1\}\.Consequently, for every coordinatekk,

∫𝒮\|mk​\(s′\)\|​𝑑ν​\(s′\)≤∫𝒮‖m​\(s′\)‖2​𝑑ν​\(s′\)≤Bm,1\.\\int\_\{\\mathcal\{S\}\}\|m\_\{k\}\(s^\{\\prime\}\)\|\\,d\\nu\(s^\{\\prime\}\)\\leq\\int\_\{\\mathcal\{S\}\}\\\|m\(s^\{\\prime\}\)\\\|\_\{2\}\\,d\\nu\(s^\{\\prime\}\)\\leq B\_\{m,1\}\.

##### Claim\.

For every stationary policyπ\\pi, there existswπ∈ℝdw^\{\\pi\}\\in\\mathbb\{R\}^\{d\}such that

Qπ​\(s,a\)=ψ​\(s,a\)⊤​wπ,Q^\{\\pi\}\(s,a\)=\\psi\(s,a\)^\{\\top\}w^\{\\pi\},where

wkπ=θr,k\+γ​θkπ,θkπ:=∫𝒮mk​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\.w^\{\\pi\}\_\{k\}=\\theta\_\{r,k\}\+\\gamma\\theta^\{\\pi\}\_\{k\},\\qquad\\theta^\{\\pi\}\_\{k\}:=\\int\_\{\\mathcal\{S\}\}m\_\{k\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\.Moreover,

‖wπ‖2≤‖θr‖2\+γ​Bm,1​Rmax1−γ\.\\\|w^\{\\pi\}\\\|\_\{2\}\\leq\\\|\\theta\_\{r\}\\\|\_\{2\}\+\\gamma\\,\\frac\{B\_\{m,1\}R\_\{\\max\}\}\{1\-\\gamma\}\.
###### Proof\.

Fix an arbitrary stationary policyπ\\pi\. We prove the result for this fixedπ\\pi\.

First, by the definition of the discounted value function,

Vπ\(s\)=𝔼π\[∑t=0∞γtr\(st,at\)\|s0=s\]\.V^\{\\pi\}\(s\)=\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{0\}=s\\right\]\.Since\|r​\(s,a\)\|≤Rmax\|r\(s,a\)\|\\leq R\_\{\\max\}for all\(s,a\)\(s,a\)and0≤γ<10\\leq\\gamma<1, we have, for everyss,

\|Vπ​\(s\)\|\\displaystyle\|V^\{\\pi\}\(s\)\|=\|𝔼π\[∑t=0∞γtr\(st,at\)\|s0=s\]\|\\displaystyle=\\left\|\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{0\}=s\\right\]\\right\|≤𝔼π\[\|∑t=0∞γtr\(st,at\)\|\|s0=s\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\left\|\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\|\\,\\middle\|\\,s\_\{0\}=s\\right\]≤𝔼π\[∑t=0∞γt\|r\(st,at\)\|\|s0=s\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\|r\(s\_\{t\},a\_\{t\}\)\|\\,\\middle\|\\,s\_\{0\}=s\\right\]≤𝔼π\[∑t=0∞γtRmax\|s0=s\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{\\max\}\\,\\middle\|\\,s\_\{0\}=s\\right\]=Rmax​∑t=0∞γt=Rmax1−γ\.\\displaystyle=R\_\{\\max\}\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}=\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\.Therefore,

‖Vπ‖∞≤Rmax1−γ\.\\\|V^\{\\pi\}\\\|\_\{\\infty\}\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\.\(10\)
The Bellman equation forQπQ^\{\\pi\}is

Qπ​\(s,a\)=r​\(s,a\)\+γ​∫𝒮P​\(s′∣s,a\)​Vπ​\(s′\)​𝑑ν​\(s′\)\.Q^\{\\pi\}\(s,a\)=r\(s,a\)\+\\gamma\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\.\(11\)We now expand the transition term in \([11](https://arxiv.org/html/2607.13498#A3.E11)\)\. Substituting the CP density representation \([9](https://arxiv.org/html/2607.13498#A3.E9)\) gives

∫𝒮P​\(s′∣s,a\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∫𝒮\(∑k=1dϕs,k​\(s\)​ϕa,k​\(a\)​mk​\(s′\)\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle=\\int\_\{\\mathcal\{S\}\}\\left\(\\sum\_\{k=1\}^\{d\}\\phi\_\{s,k\}\(s\)\\phi\_\{a,k\}\(a\)m\_\{k\}\(s^\{\\prime\}\)\\right\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∫𝒮\(∑k=1dψk​\(s,a\)​mk​\(s′\)\)​Vπ​\(s′\)​𝑑ν​\(s′\)\.\\displaystyle=\\int\_\{\\mathcal\{S\}\}\\left\(\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)m\_\{k\}\(s^\{\\prime\}\)\\right\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\.\(12\)
Before moving the finite sum outside the integral, we check that each term is integrable\. For a fixed coordinatekk, using \([10](https://arxiv.org/html/2607.13498#A3.E10)\),

∫𝒮\|mk​\(s′\)​Vπ​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle\\int\_\{\\mathcal\{S\}\}\\left\|m\_\{k\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\right\|\\,d\\nu\(s^\{\\prime\}\)≤∫𝒮\|mk​\(s′\)\|​\|Vπ​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle\\leq\\int\_\{\\mathcal\{S\}\}\|m\_\{k\}\(s^\{\\prime\}\)\|\\,\|V^\{\\pi\}\(s^\{\\prime\}\)\|\\,d\\nu\(s^\{\\prime\}\)≤Rmax1−γ​∫𝒮\|mk​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\\int\_\{\\mathcal\{S\}\}\|m\_\{k\}\(s^\{\\prime\}\)\|\\,d\\nu\(s^\{\\prime\}\)≤Rmax1−γ​Bm,1<∞\.\\displaystyle\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}B\_\{m,1\}<\\infty\.Thusmk​Vπm\_\{k\}V^\{\\pi\}is integrable for everykk, and the quantity

θkπ:=∫𝒮mk​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\theta^\{\\pi\}\_\{k\}:=\\int\_\{\\mathcal\{S\}\}m\_\{k\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)is well\-defined\.

Since the sum overkkcontains only finitely many terms, exchanging the sum and the integral uses only the elementary linearity of the integral\. Therefore,

∫𝒮\(∑k=1dψk​\(s,a\)​mk​\(s′\)\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle\\int\_\{\\mathcal\{S\}\}\\left\(\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)m\_\{k\}\(s^\{\\prime\}\)\\right\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∫𝒮∑k=1dψk​\(s,a\)​mk​\(s′\)​Vπ​\(s′\)​d​ν​\(s′\)\\displaystyle=\\int\_\{\\mathcal\{S\}\}\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)m\_\{k\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∑k=1d∫𝒮ψk​\(s,a\)​mk​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle=\\sum\_\{k=1\}^\{d\}\\int\_\{\\mathcal\{S\}\}\\psi\_\{k\}\(s,a\)m\_\{k\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∑k=1dψk​\(s,a\)​∫𝒮mk​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\int\_\{\\mathcal\{S\}\}m\_\{k\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∑k=1dψk​\(s,a\)​θkπ\.\\displaystyle=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\theta^\{\\pi\}\_\{k\}\.\(13\)In the third line above,ψk​\(s,a\)\\psi\_\{k\}\(s,a\)is taken outside the integral because, for fixed\(s,a\)\(s,a\), it is a scalar that does not depend ons′s^\{\\prime\}\.

Next, we expand the reward term\. Sincer​\(s,a\)=ψ​\(s,a\)⊤​θrr\(s,a\)=\\psi\(s,a\)^\{\\top\}\\theta\_\{r\}, we have

r​\(s,a\)=∑k=1dψk​\(s,a\)​θr,k=∑k=1dϕs,k​\(s\)​ϕa,k​\(a\)​θr,k\.r\(s,a\)=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\theta\_\{r,k\}=\\sum\_\{k=1\}^\{d\}\\phi\_\{s,k\}\(s\)\\phi\_\{a,k\}\(a\)\\theta\_\{r,k\}\.\(14\)
Combining \([11](https://arxiv.org/html/2607.13498#A3.E11)\), \([13](https://arxiv.org/html/2607.13498#A3.E13)\), and \([14](https://arxiv.org/html/2607.13498#A3.E14)\), we obtain

Qπ​\(s,a\)\\displaystyle Q^\{\\pi\}\(s,a\)=r​\(s,a\)\+γ​∫𝒮P​\(s′∣s,a\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle=r\(s,a\)\+\\gamma\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∑k=1dψk​\(s,a\)​θr,k\+γ​∑k=1dψk​\(s,a\)​θkπ\\displaystyle=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\theta\_\{r,k\}\+\\gamma\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\theta^\{\\pi\}\_\{k\}=∑k=1dψk​\(s,a\)​θr,k\+∑k=1dψk​\(s,a\)​γ​θkπ\\displaystyle=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\theta\_\{r,k\}\+\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\gamma\\theta^\{\\pi\}\_\{k\}=∑k=1dψk​\(s,a\)​\(θr,k\+γ​θkπ\)\.\\displaystyle=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\left\(\\theta\_\{r,k\}\+\\gamma\\theta^\{\\pi\}\_\{k\}\\right\)\.Definewπ∈ℝdw^\{\\pi\}\\in\\mathbb\{R\}^\{d\}coordinatewise by

wkπ:=θr,k\+γ​θkπ,k=1,…,d\.w^\{\\pi\}\_\{k\}:=\\theta\_\{r,k\}\+\\gamma\\theta^\{\\pi\}\_\{k\},\\qquad k=1,\\ldots,d\.Then the previous display becomes

Qπ​\(s,a\)=∑k=1dψk​\(s,a\)​wkπ=ψ​\(s,a\)⊤​wπ\.Q^\{\\pi\}\(s,a\)=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)w^\{\\pi\}\_\{k\}=\\psi\(s,a\)^\{\\top\}w^\{\\pi\}\.This proves the claimed linear representation ofQπQ^\{\\pi\}\.

It remains to prove the stated norm bound\. By definition,

θπ=\(∫𝒮m1​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\),…,∫𝒮md​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\)⊤\.\\theta^\{\\pi\}=\\left\(\\int\_\{\\mathcal\{S\}\}m\_\{1\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\),\\ldots,\\int\_\{\\mathcal\{S\}\}m\_\{d\}\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\\right\)^\{\\top\}\.Equivalently, in vector notation,

θπ=∫𝒮m​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\),\\theta^\{\\pi\}=\\int\_\{\\mathcal\{S\}\}m\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\),where the vector integral is understood coordinatewise\.

We show that

‖θπ‖2≤Bm,1​Rmax1−γ\.\\\|\\theta^\{\\pi\}\\\|\_\{2\}\\leq\\frac\{B\_\{m,1\}R\_\{\\max\}\}\{1\-\\gamma\}\.Ifθπ=0\\theta^\{\\pi\}=0, this inequality is immediate\. Otherwise, let

u=θπ‖θπ‖2\.u=\\frac\{\\theta^\{\\pi\}\}\{\\\|\\theta^\{\\pi\}\\\|\_\{2\}\}\.Then‖u‖2=1\\\|u\\\|\_\{2\}=1, and hence

‖θπ‖2\\displaystyle\\\|\\theta^\{\\pi\}\\\|\_\{2\}=u⊤​θπ\\displaystyle=u^\{\\top\}\\theta^\{\\pi\}=u⊤​∫𝒮m​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle=u^\{\\top\}\\int\_\{\\mathcal\{S\}\}m\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=∫𝒮u⊤​m​\(s′\)​Vπ​\(s′\)​𝑑ν​\(s′\)\\displaystyle=\\int\_\{\\mathcal\{S\}\}u^\{\\top\}m\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)≤∫𝒮\|u⊤​m​\(s′\)​Vπ​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle\\leq\\int\_\{\\mathcal\{S\}\}\\left\|u^\{\\top\}m\(s^\{\\prime\}\)V^\{\\pi\}\(s^\{\\prime\}\)\\right\|\\,d\\nu\(s^\{\\prime\}\)=∫𝒮\|u⊤​m​\(s′\)\|​\|Vπ​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle=\\int\_\{\\mathcal\{S\}\}\\left\|u^\{\\top\}m\(s^\{\\prime\}\)\\right\|\\left\|V^\{\\pi\}\(s^\{\\prime\}\)\\right\|\\,d\\nu\(s^\{\\prime\}\)≤∫𝒮‖u‖2​‖m​\(s′\)‖2​\|Vπ​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle\\leq\\int\_\{\\mathcal\{S\}\}\\\|u\\\|\_\{2\}\\\|m\(s^\{\\prime\}\)\\\|\_\{2\}\\left\|V^\{\\pi\}\(s^\{\\prime\}\)\\right\|\\,d\\nu\(s^\{\\prime\}\)=∫𝒮‖m​\(s′\)‖2​\|Vπ​\(s′\)\|​𝑑ν​\(s′\)\\displaystyle=\\int\_\{\\mathcal\{S\}\}\\\|m\(s^\{\\prime\}\)\\\|\_\{2\}\\left\|V^\{\\pi\}\(s^\{\\prime\}\)\\right\|\\,d\\nu\(s^\{\\prime\}\)≤Rmax1−γ​∫𝒮‖m​\(s′\)‖2​𝑑ν​\(s′\)\\displaystyle\\leq\\frac\{R\_\{\\max\}\}\{1\-\\gamma\}\\int\_\{\\mathcal\{S\}\}\\\|m\(s^\{\\prime\}\)\\\|\_\{2\}\\,d\\nu\(s^\{\\prime\}\)≤Bm,1​Rmax1−γ\.\\displaystyle\\leq\\frac\{B\_\{m,1\}R\_\{\\max\}\}\{1\-\\gamma\}\.The inequality

\|u⊤​m​\(s′\)\|≤‖u‖2​‖m​\(s′\)‖2\|u^\{\\top\}m\(s^\{\\prime\}\)\|\\leq\\\|u\\\|\_\{2\}\\\|m\(s^\{\\prime\}\)\\\|\_\{2\}is the Cauchy–Schwarz inequality\. The key point is that this argument bounds the Euclidean norm of the whole vectorθπ\\theta^\{\\pi\}directly, and therefore does not introduce an unnecessaryd\\sqrt\{d\}factor\.

Finally, since

wπ=θr\+γ​θπ,w^\{\\pi\}=\\theta\_\{r\}\+\\gamma\\theta^\{\\pi\},the triangle inequality gives

‖wπ‖2\\displaystyle\\\|w^\{\\pi\}\\\|\_\{2\}=‖θr\+γ​θπ‖2\\displaystyle=\\\|\\theta\_\{r\}\+\\gamma\\theta^\{\\pi\}\\\|\_\{2\}≤‖θr‖2\+‖γ​θπ‖2\\displaystyle\\leq\\\|\\theta\_\{r\}\\\|\_\{2\}\+\\\|\\gamma\\theta^\{\\pi\}\\\|\_\{2\}=‖θr‖2\+γ​‖θπ‖2\\displaystyle=\\\|\\theta\_\{r\}\\\|\_\{2\}\+\\gamma\\\|\\theta^\{\\pi\}\\\|\_\{2\}≤‖θr‖2\+γ​Bm,1​Rmax1−γ\.\\displaystyle\\leq\\\|\\theta\_\{r\}\\\|\_\{2\}\+\\gamma\\,\\frac\{B\_\{m,1\}R\_\{\\max\}\}\{1\-\\gamma\}\.This proves the claimed norm bound and completes the proof\. ∎

##### Remark \(Linear\-MDP structure and feature bound\)

Defining signed measuresμk​\(A\):=∫Amk​\(s′\)​𝑑ν​\(s′\)\\mu\_\{k\}\(A\):=\\int\_\{A\}m\_\{k\}\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\), the CP transition density is equivalent toP\(⋅∣s,a\)=∑k=1dψk\(s,a\)μk\(⋅\)P\(\\cdot\\mid s,a\)=\\sum\_\{k=1\}^\{d\}\\psi\_\{k\}\(s,a\)\\mu\_\{k\}\(\\cdot\)\. Together withr​\(s,a\)=ψ​\(s,a\)⊤​θrr\(s,a\)=\\psi\(s,a\)^\{\\top\}\\theta\_\{r\}, this matches the linear\-MDP definition ofJinet al\.\[[2020](https://arxiv.org/html/2607.13498#bib.bib15), Definition 2\]with feature mapψ\\psi, used in Appendix[C\.7](https://arxiv.org/html/2607.13498#A3.SS7)\. The same swap\-of\-sum\-and\-integral argument as the proof of Q\-linearity above gives, for any bounded measurablef:𝒮→ℝf:\\mathcal\{S\}\\to\\mathbb\{R\},∫𝒮P​\(s′∣s,a\)​f​\(s′\)​𝑑ν​\(s′\)=ψ​\(s,a\)⊤​θf\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)f\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)=\\psi\(s,a\)^\{\\top\}\\theta\_\{f\}with\(θf\)k:=∫𝒮mk​\(s′\)​f​\(s′\)​𝑑ν​\(s′\)\(\\theta\_\{f\}\)\_\{k\}:=\\int\_\{\\mathcal\{S\}\}m\_\{k\}\(s^\{\\prime\}\)f\(s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)and‖θf‖2≤Bm,1​‖f‖∞\\\|\\theta\_\{f\}\\\|\_\{2\}\\leq B\_\{m,1\}\\\|f\\\|\_\{\\infty\}\. Finally, the Hadamard\-product inequality‖u⊙v‖2≤‖u‖2​‖v‖2\\\|u\\odot v\\\|\_\{2\}\\leq\\\|u\\\|\_\{2\}\\\|v\\\|\_\{2\}applied toψ=ϕs⊙ϕa\\psi=\\phi\_\{s\}\\odot\\phi\_\{a\}gives the uniform feature bound‖ψ​\(s,a\)‖2≤Bϕ2\\\|\\psi\(s,a\)\\\|\_\{2\}\\leq B\_\{\\phi\}^\{2\}\.

### C\.3Proof of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1): Additive Covering Number Decomposition

This is the main technical step\. The Hadamard product structure converts the joint covering of𝒢fac\\mathcal\{G\}\_\{\\mathrm\{fac\}\}into a product of three independent coverings on the state, action, and next\-state components\. After taking logarithms, this product becomes an additive decomposition of metric entropies\. Without using this decomposition, a generic uniform\-convergence treatment of the state\-action pair would cover the whole state\-action feature class directly and would therefore give the same\(ds\+da\)\(d\_\{s\}\+d\_\{a\}\)\-dimensional entropy scaling as the joint class\. The Hadamard structure reduces the factored upper bound to the largest of the state and action dimensions\.

Throughout this appendix, covering numbers for the component classes are taken with respect to the uniform sup\-ℓ2\\ell\_\{2\}metrics

ds​\(ϕs,ϕ~s\):=sups∈𝒮‖ϕs​\(s\)−ϕ~s​\(s\)‖2,d\_\{s\}\(\\phi\_\{s\},\\widetilde\{\\phi\}\_\{s\}\):=\\sup\_\{s\\in\\mathcal\{S\}\}\\\|\\phi\_\{s\}\(s\)\-\\widetilde\{\\phi\}\_\{s\}\(s\)\\\|\_\{2\},da​\(ϕa,ϕ~a\):=supa∈𝒜‖ϕa​\(a\)−ϕ~a​\(a\)‖2,d\_\{a\}\(\\phi\_\{a\},\\widetilde\{\\phi\}\_\{a\}\):=\\sup\_\{a\\in\\mathcal\{A\}\}\\\|\\phi\_\{a\}\(a\)\-\\widetilde\{\\phi\}\_\{a\}\(a\)\\\|\_\{2\},and

dm​\(m,m~\):=sups′∈𝒮‖m​\(s′\)−m~​\(s′\)‖2\.d\_\{m\}\(m,\\widetilde\{m\}\):=\\sup\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\\|m\(s^\{\\prime\}\)\-\\widetilde\{m\}\(s^\{\\prime\}\)\\\|\_\{2\}\.For scoring functionsg:𝒮×𝒜×𝒮→ℝg:\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\to\\mathbb\{R\}, the covering number is taken with respect to the usual uniform norm

‖g−g~‖∞:=sup\(s,a,s′\)∈𝒮×𝒜×𝒮\|g​\(s,a,s′\)−g~​\(s,a,s′\)\|\.\\\|g\-\\widetilde\{g\}\\\|\_\{\\infty\}:=\\sup\_\{\(s,a,s^\{\\prime\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\}\|g\(s,a,s^\{\\prime\}\)\-\\widetilde\{g\}\(s,a,s^\{\\prime\}\)\|\.
###### Theorem\(Restatement of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\)\.

1. *\(i\)**Additive decomposition\.*For everyϵ\>0\\epsilon\>0, log⁡𝒩​\(𝒢fac,ϵ\)≤log⁡𝒩​\(ℱs,ϵ3​Bm,∞​Bϕ\)\+log⁡𝒩​\(ℱa,ϵ3​Bm,∞​Bϕ\)\+log⁡𝒩​\(ℱm,ϵ3​Bϕ2\)\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\leq\\log\\mathcal\{N\}\\\!\\Bigl\(\\mathcal\{F\}\_\{s\},\\tfrac\{\\epsilon\}\{3B\_\{m,\\infty\}B\_\{\\phi\}\}\\Bigr\)\+\\log\\mathcal\{N\}\\\!\\Bigl\(\\mathcal\{F\}\_\{a\},\\tfrac\{\\epsilon\}\{3B\_\{m,\\infty\}B\_\{\\phi\}\}\\Bigr\)\+\\log\\mathcal\{N\}\\\!\\Bigl\(\\mathcal\{F\}\_\{m\},\\tfrac\{\\epsilon\}\{3B\_\{\\phi\}^\{2\}\}\\Bigr\)\.\(15\)
2. *\(ii\)**Containment\.*If ℱs​a⊇\{ϕs⊙ϕa:ϕs∈ℱs,ϕa∈ℱa\},\\mathcal\{F\}\_\{sa\}\\supseteq\\\{\\phi\_\{s\}\\odot\\phi\_\{a\}:\\phi\_\{s\}\\in\\mathcal\{F\}\_\{s\},\\phi\_\{a\}\\in\\mathcal\{F\}\_\{a\}\\\},then𝒢fac⊆𝒢joint\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\mathrm\{joint\}\}, hence 𝒩​\(𝒢fac,ϵ\)≤𝒩​\(𝒢joint,ϵ\)\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\leq\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\epsilon\)for everyϵ\>0\\epsilon\>0\.

###### Lemma C\.1\(Perturbation bound for factored CP scores\)\.

Let

g​\(s,a,s′\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)g\(s,a,s^\{\\prime\}\)=\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\)and

g~​\(s,a,s′\)=\(ϕ~s​\(s\)⊙ϕ~a​\(a\)\)⊤​m~​\(s′\)\.\\widetilde\{g\}\(s,a,s^\{\\prime\}\)=\(\\widetilde\{\\phi\}\_\{s\}\(s\)\\odot\\widetilde\{\\phi\}\_\{a\}\(a\)\)^\{\\top\}\\widetilde\{m\}\(s^\{\\prime\}\)\.Assume

sups‖ϕs​\(s\)‖2,sups‖ϕ~s​\(s\)‖2≤Bϕ,supa‖ϕa​\(a\)‖2,supa‖ϕ~a​\(a\)‖2≤Bϕ,\\sup\_\{s\}\\\|\\phi\_\{s\}\(s\)\\\|\_\{2\},\\ \\sup\_\{s\}\\\|\\widetilde\{\\phi\}\_\{s\}\(s\)\\\|\_\{2\}\\leq B\_\{\\phi\},\\qquad\\sup\_\{a\}\\\|\\phi\_\{a\}\(a\)\\\|\_\{2\},\\ \\sup\_\{a\}\\\|\\widetilde\{\\phi\}\_\{a\}\(a\)\\\|\_\{2\}\\leq B\_\{\\phi\},and

sups′‖m~​\(s′\)‖2≤Bm,∞\.\\sup\_\{s^\{\\prime\}\}\\\|\\widetilde\{m\}\(s^\{\\prime\}\)\\\|\_\{2\}\\leq B\_\{m,\\infty\}\.Define

Δs:=sups‖ϕs​\(s\)−ϕ~s​\(s\)‖2,Δa:=supa‖ϕa​\(a\)−ϕ~a​\(a\)‖2,\\Delta\_\{s\}:=\\sup\_\{s\}\\\|\\phi\_\{s\}\(s\)\-\\widetilde\{\\phi\}\_\{s\}\(s\)\\\|\_\{2\},\\qquad\\Delta\_\{a\}:=\\sup\_\{a\}\\\|\\phi\_\{a\}\(a\)\-\\widetilde\{\\phi\}\_\{a\}\(a\)\\\|\_\{2\},and

Δm:=sups′‖m​\(s′\)−m~​\(s′\)‖2\.\\Delta\_\{m\}:=\\sup\_\{s^\{\\prime\}\}\\\|m\(s^\{\\prime\}\)\-\\widetilde\{m\}\(s^\{\\prime\}\)\\\|\_\{2\}\.Then

‖g−g~‖∞≤Bϕ2​Δm\+Bm,∞​Bϕ​Δs\+Bm,∞​Bϕ​Δa\.\\\|g\-\\widetilde\{g\}\\\|\_\{\\infty\}\\leq B\_\{\\phi\}^\{2\}\\Delta\_\{m\}\+B\_\{m,\\infty\}B\_\{\\phi\}\\Delta\_\{s\}\+B\_\{m,\\infty\}B\_\{\\phi\}\\Delta\_\{a\}\.\(16\)

###### Proof\.

Write

x​\(s,a\):=ϕs​\(s\)⊙ϕa​\(a\),x~​\(s,a\):=ϕ~s​\(s\)⊙ϕ~a​\(a\)\.x\(s,a\):=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\),\\qquad\\widetilde\{x\}\(s,a\):=\\widetilde\{\\phi\}\_\{s\}\(s\)\\odot\\widetilde\{\\phi\}\_\{a\}\(a\)\.For fixed\(s,a,s′\)\(s,a,s^\{\\prime\}\),

g​\(s,a,s′\)−g~​\(s,a,s′\)=x​\(s,a\)⊤​\(m​\(s′\)−m~​\(s′\)\)\+\(x​\(s,a\)−x~​\(s,a\)\)⊤​m~​\(s′\)\.g\(s,a,s^\{\\prime\}\)\-\\widetilde\{g\}\(s,a,s^\{\\prime\}\)=x\(s,a\)^\{\\top\}\\bigl\(m\(s^\{\\prime\}\)\-\\widetilde\{m\}\(s^\{\\prime\}\)\\bigr\)\+\\bigl\(x\(s,a\)\-\\widetilde\{x\}\(s,a\)\\bigr\)^\{\\top\}\\widetilde\{m\}\(s^\{\\prime\}\)\.By Cauchy–Schwarz,

\|g​\(s,a,s′\)−g~​\(s,a,s′\)\|≤‖x​\(s,a\)‖2​‖m​\(s′\)−m~​\(s′\)‖2\+‖x​\(s,a\)−x~​\(s,a\)‖2​‖m~​\(s′\)‖2\.\|g\(s,a,s^\{\\prime\}\)\-\\widetilde\{g\}\(s,a,s^\{\\prime\}\)\|\\leq\\\|x\(s,a\)\\\|\_\{2\}\\\|m\(s^\{\\prime\}\)\-\\widetilde\{m\}\(s^\{\\prime\}\)\\\|\_\{2\}\+\\\|x\(s,a\)\-\\widetilde\{x\}\(s,a\)\\\|\_\{2\}\\\|\\widetilde\{m\}\(s^\{\\prime\}\)\\\|\_\{2\}\.The Hadamard product bound gives

‖x​\(s,a\)‖2=‖ϕs​\(s\)⊙ϕa​\(a\)‖2≤‖ϕs​\(s\)‖2​‖ϕa​\(a\)‖2≤Bϕ2\.\\\|x\(s,a\)\\\|\_\{2\}=\\\|\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\\\|\_\{2\}\\leq\\\|\\phi\_\{s\}\(s\)\\\|\_\{2\}\\\|\\phi\_\{a\}\(a\)\\\|\_\{2\}\\leq B\_\{\\phi\}^\{2\}\.Moreover,

x​\(s,a\)−x~​\(s,a\)=\(ϕs​\(s\)−ϕ~s​\(s\)\)⊙ϕa​\(a\)\+ϕ~s​\(s\)⊙\(ϕa​\(a\)−ϕ~a​\(a\)\)\.x\(s,a\)\-\\widetilde\{x\}\(s,a\)=\(\\phi\_\{s\}\(s\)\-\\widetilde\{\\phi\}\_\{s\}\(s\)\)\\odot\\phi\_\{a\}\(a\)\+\\widetilde\{\\phi\}\_\{s\}\(s\)\\odot\(\\phi\_\{a\}\(a\)\-\\widetilde\{\\phi\}\_\{a\}\(a\)\)\.Using

‖u⊙v‖2≤‖v‖∞​‖u‖2≤‖v‖2​‖u‖2,\\\|u\\odot v\\\|\_\{2\}\\leq\\\|v\\\|\_\{\\infty\}\\\|u\\\|\_\{2\}\\leq\\\|v\\\|\_\{2\}\\\|u\\\|\_\{2\},we obtain

‖x​\(s,a\)−x~​\(s,a\)‖2≤Bϕ​‖ϕs​\(s\)−ϕ~s​\(s\)‖2\+Bϕ​‖ϕa​\(a\)−ϕ~a​\(a\)‖2\.\\\|x\(s,a\)\-\\widetilde\{x\}\(s,a\)\\\|\_\{2\}\\leq B\_\{\\phi\}\\\|\\phi\_\{s\}\(s\)\-\\widetilde\{\\phi\}\_\{s\}\(s\)\\\|\_\{2\}\+B\_\{\\phi\}\\\|\\phi\_\{a\}\(a\)\-\\widetilde\{\\phi\}\_\{a\}\(a\)\\\|\_\{2\}\.Taking the supremum over\(s,a,s′\)\(s,a,s^\{\\prime\}\)proves \([16](https://arxiv.org/html/2607.13498#A3.E16)\)\. ∎

###### Proof\.

We prove the two statements separately\.

##### Part \(i\): additive decomposition\.

Choose component coversℱ^s\\widehat\{\\mathcal\{F\}\}\_\{s\},ℱ^a\\widehat\{\\mathcal\{F\}\}\_\{a\},ℱ^m\\widehat\{\\mathcal\{F\}\}\_\{m\}ofℱs,ℱa,ℱm\\mathcal\{F\}\_\{s\},\\mathcal\{F\}\_\{a\},\\mathcal\{F\}\_\{m\}at radii

ϵs:=ϵ3​Bm,∞​Bϕ,ϵa:=ϵ3​Bm,∞​Bϕ,ϵm:=ϵ3​Bϕ2,\\epsilon\_\{s\}:=\\frac\{\\epsilon\}\{3B\_\{m,\\infty\}B\_\{\\phi\}\},\\qquad\\epsilon\_\{a\}:=\\frac\{\\epsilon\}\{3B\_\{m,\\infty\}B\_\{\\phi\}\},\\qquad\\epsilon\_\{m\}:=\\frac\{\\epsilon\}\{3B\_\{\\phi\}^\{2\}\},respectively, under the corresponding uniform sup\-ℓ2\\ell\_\{2\}metrics\. Take any

g​\(s,a,s′\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)∈𝒢fac\.g\(s,a,s^\{\\prime\}\)=\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\)\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\.By the definitions of the three covers, choose

ϕ^s∈ℱ^s,ϕ^a∈ℱ^a,m^∈ℱ^m\\widehat\{\\phi\}\_\{s\}\\in\\widehat\{\\mathcal\{F\}\}\_\{s\},\\qquad\\widehat\{\\phi\}\_\{a\}\\in\\widehat\{\\mathcal\{F\}\}\_\{a\},\\qquad\\widehat\{m\}\\in\\widehat\{\\mathcal\{F\}\}\_\{m\}such that

sups‖ϕs​\(s\)−ϕ^s​\(s\)‖2≤ϵs,supa‖ϕa​\(a\)−ϕ^a​\(a\)‖2≤ϵa,sups′‖m​\(s′\)−m^​\(s′\)‖2≤ϵm\.\\sup\_\{s\}\\\|\\phi\_\{s\}\(s\)\-\\widehat\{\\phi\}\_\{s\}\(s\)\\\|\_\{2\}\\leq\\epsilon\_\{s\},\\qquad\\sup\_\{a\}\\\|\\phi\_\{a\}\(a\)\-\\widehat\{\\phi\}\_\{a\}\(a\)\\\|\_\{2\}\\leq\\epsilon\_\{a\},\\qquad\\sup\_\{s^\{\\prime\}\}\\\|m\(s^\{\\prime\}\)\-\\widehat\{m\}\(s^\{\\prime\}\)\\\|\_\{2\}\\leq\\epsilon\_\{m\}\.Define

g^​\(s,a,s′\)=\(ϕ^s​\(s\)⊙ϕ^a​\(a\)\)⊤​m^​\(s′\)\.\\widehat\{g\}\(s,a,s^\{\\prime\}\)=\(\\widehat\{\\phi\}\_\{s\}\(s\)\\odot\\widehat\{\\phi\}\_\{a\}\(a\)\)^\{\\top\}\\widehat\{m\}\(s^\{\\prime\}\)\.Applying Lemma[C\.1](https://arxiv.org/html/2607.13498#A3.Thmtheorem1)gives

‖g−g^‖∞\\displaystyle\\\|g\-\\widehat\{g\}\\\|\_\{\\infty\}≤Bϕ2​ϵm\+Bm,∞​Bϕ​ϵs\+Bm,∞​Bϕ​ϵa\\displaystyle\\leq B\_\{\\phi\}^\{2\}\\epsilon\_\{m\}\+B\_\{m,\\infty\}B\_\{\\phi\}\\epsilon\_\{s\}\+B\_\{m,\\infty\}B\_\{\\phi\}\\epsilon\_\{a\}=ϵ3\+ϵ3\+ϵ3=ϵ\.\\displaystyle=\\frac\{\\epsilon\}\{3\}\+\\frac\{\\epsilon\}\{3\}\+\\frac\{\\epsilon\}\{3\}=\\epsilon\.Thus the product class

𝒢^fac:=\{\(s,a,s′\)↦\(ϕ^s​\(s\)⊙ϕ^a​\(a\)\)⊤​m^​\(s′\):ϕ^s∈ℱ^s,ϕ^a∈ℱ^a,m^∈ℱ^m\}\\widehat\{\\mathcal\{G\}\}\_\{\\mathrm\{fac\}\}:=\\left\\\{\(s,a,s^\{\\prime\}\)\\mapsto\(\\widehat\{\\phi\}\_\{s\}\(s\)\\odot\\widehat\{\\phi\}\_\{a\}\(a\)\)^\{\\top\}\\widehat\{m\}\(s^\{\\prime\}\):\\widehat\{\\phi\}\_\{s\}\\in\\widehat\{\\mathcal\{F\}\}\_\{s\},\\ \\widehat\{\\phi\}\_\{a\}\\in\\widehat\{\\mathcal\{F\}\}\_\{a\},\\ \\widehat\{m\}\\in\\widehat\{\\mathcal\{F\}\}\_\{m\}\\right\\\}is anϵ\\epsilon\-cover of𝒢fac\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\. Each element of𝒢^fac\\widehat\{\\mathcal\{G\}\}\_\{\\mathrm\{fac\}\}is determined by a triple from the three component covers, so

\|𝒢^fac\|≤\|ℱ^s\|​\|ℱ^a\|​\|ℱ^m\|\.\|\\widehat\{\\mathcal\{G\}\}\_\{\\mathrm\{fac\}\}\|\\leq\|\\widehat\{\\mathcal\{F\}\}\_\{s\}\|\\,\|\\widehat\{\\mathcal\{F\}\}\_\{a\}\|\\,\|\\widehat\{\\mathcal\{F\}\}\_\{m\}\|\.Taking the smallest possible component covers, or equivalently taking the infimum over such covers, yields

𝒩​\(𝒢fac,ϵ\)≤𝒩​\(ℱs,ϵs\)​𝒩​\(ℱa,ϵa\)​𝒩​\(ℱm,ϵm\)\.\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\leq\\mathcal\{N\}\(\\mathcal\{F\}\_\{s\},\\epsilon\_\{s\}\)\\mathcal\{N\}\(\\mathcal\{F\}\_\{a\},\\epsilon\_\{a\}\)\\mathcal\{N\}\(\\mathcal\{F\}\_\{m\},\\epsilon\_\{m\}\)\.Taking logarithms and substituting the definitions ofϵs,ϵa,ϵm\\epsilon\_\{s\},\\epsilon\_\{a\},\\epsilon\_\{m\}gives \([15](https://arxiv.org/html/2607.13498#A3.E15)\)\.

##### Part \(ii\): containment\.

Take anyg∈𝒢facg\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\. Then, for someϕs∈ℱs\\phi\_\{s\}\\in\\mathcal\{F\}\_\{s\},ϕa∈ℱa\\phi\_\{a\}\\in\\mathcal\{F\}\_\{a\}, andm∈ℱmm\\in\\mathcal\{F\}\_\{m\},

g​\(s,a,s′\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)\.g\(s,a,s^\{\\prime\}\)=\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\)\.By the assumption onℱs​a\\mathcal\{F\}\_\{sa\},

φ​\(s,a\):=ϕs​\(s\)⊙ϕa​\(a\)\\varphi\(s,a\):=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)belongs toℱs​a\\mathcal\{F\}\_\{sa\}\. Takingμ:=m\\mu:=m, we get

g​\(s,a,s′\)=φ​\(s,a\)⊤​μ​\(s′\),g\(s,a,s^\{\\prime\}\)=\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\),sog∈𝒢jointg\\in\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\. Hence

𝒢fac⊆𝒢joint\.\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\.The covering\-number inequality follows because anyϵ\\epsilon\-cover of a set also covers all of its subsets\. ∎

### C\.4Lipschitz Instantiation: an Upper\-Bound vs\. Upper\-Bound Comparison

We now explain how the additive decomposition in \([15](https://arxiv.org/html/2607.13498#A3.E15)\) translates into the stated entropy scaling under standard Lipschitz covering bounds\. Assume that

𝒮⊆ℝds,𝒜⊆ℝda\\mathcal\{S\}\\subseteq\\mathbb\{R\}^\{d\_\{s\}\},\\qquad\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{d\_\{a\}\}are compact, and thatℱs,ℱa,ℱm\\mathcal\{F\}\_\{s\},\\mathcal\{F\}\_\{a\},\\mathcal\{F\}\_\{m\}are uniformly bounded Lipschitz classes\. ForLL\-Lipschitz maps from a compact setX⊆ℝpX\\subseteq\\mathbb\{R\}^\{p\}into a bounded subset ofℝd\\mathbb\{R\}^\{d\}, the standard entropy bound gives

log⁡𝒩​\(ℱ,ϵ\)≤c​d​\(L​diam⁡\(X\)/ϵ\)p\\log\\mathcal\{N\}\(\\mathcal\{F\},\\epsilon\)\\leq c\\,d\\bigl\(L\\operatorname\{diam\}\(X\)/\\epsilon\\bigr\)^\{p\}for the relevant range ofϵ\\epsilon; seeWainwright \[[2019](https://arxiv.org/html/2607.13498#bib.bib25), Theorem 5\.7\]\. The important point is that the exponent is the dimensionppof the input domain\. Applying this bound to the three terms in \([15](https://arxiv.org/html/2607.13498#A3.E15)\) yields finite constantsAs,Aa,AmA\_\{s\},A\_\{a\},A\_\{m\}such that

log⁡𝒩​\(𝒢fac,ϵ\)≤As​ϵ−ds\+Aa​ϵ−da\+Am​ϵ−ds\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\leq A\_\{s\}\\epsilon^\{\-d\_\{s\}\}\+A\_\{a\}\\epsilon^\{\-d\_\{a\}\}\+A\_\{m\}\\epsilon^\{\-d\_\{s\}\}\.Therefore, for

Dmax:=max⁡\(ds,da\)D\_\{\\max\}:=\\max\(d\_\{s\},d\_\{a\}\)and0<ϵ≤10<\\epsilon\\leq 1,

log⁡𝒩​\(𝒢fac,ϵ\)≤Afac​ϵ−Dmax,\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\leq A\_\{\\mathrm\{fac\}\}\\epsilon^\{\-D\_\{\\max\}\},\(17\)whereAfac<∞A\_\{\\mathrm\{fac\}\}<\\inftydepends on the Lipschitz constants, diameters, output dimension, and uniform radius bounds, but not onϵ\\epsilon\. For the joint class, the unrestricted state\-action encoder is a Lipschitz class on the product domain

𝒮×𝒜⊆ℝds\+da\.\\mathcal\{S\}\\times\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{d\_\{s\}\+d\_\{a\}\}\.The same two\-term covering argument for

g​\(s,a,s′\)=φ​\(s,a\)⊤​μ​\(s′\)g\(s,a,s^\{\\prime\}\)=\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\)gives

log⁡𝒩​\(𝒢joint,ϵ\)≤log⁡𝒩​\(ℱs​a,ϵ2​Bm,∞\)\+log⁡𝒩​\(ℱm,ϵ2​Bs​a\),\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\epsilon\)\\leq\\log\\mathcal\{N\}\\\!\\Bigl\(\\mathcal\{F\}\_\{sa\},\\tfrac\{\\epsilon\}\{2B\_\{m,\\infty\}\}\\Bigr\)\+\\log\\mathcal\{N\}\\\!\\Bigl\(\\mathcal\{F\}\_\{m\},\\tfrac\{\\epsilon\}\{2B\_\{sa\}\}\\Bigr\),whereBs​aB\_\{sa\}is a uniform bound on‖φ​\(s,a\)‖2\\\|\\varphi\(s,a\)\\\|\_\{2\}\. Applying the Lipschitz entropy bound toℱs​a\\mathcal\{F\}\_\{sa\}on the product domain and toℱm\\mathcal\{F\}\_\{m\}on𝒮\\mathcal\{S\}yields finite constantsAs​a,Am′A\_\{sa\},A\_\{m\}^\{\\prime\}such that

log⁡𝒩​\(𝒢joint,ϵ\)≤As​a​ϵ−\(ds\+da\)\+Am′​ϵ−ds\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\epsilon\)\\leq A\_\{sa\}\\epsilon^\{\-\(d\_\{s\}\+d\_\{a\}\)\}\+A\_\{m\}^\{\\prime\}\\epsilon^\{\-d\_\{s\}\}\.Sinceds≤ds\+dad\_\{s\}\\leq d\_\{s\}\+d\_\{a\}, for0<ϵ≤10<\\epsilon\\leq 1this implies

log⁡𝒩​\(𝒢joint,ϵ\)≤Ajoint​ϵ−\(ds\+da\)\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\epsilon\)\\leq A\_\{\\mathrm\{joint\}\}\\epsilon^\{\-\(d\_\{s\}\+d\_\{a\}\)\}\(18\)for a finite constantAjointA\_\{\\mathrm\{joint\}\}independent ofϵ\\epsilon\. Combining \([17](https://arxiv.org/html/2607.13498#A3.E17)\) and \([18](https://arxiv.org/html/2607.13498#A3.E18)\), the factored and joint upper bounds scale as

log⁡𝒩​\(𝒢fac,ϵ\)≲ϵ−max⁡\(ds,da\),log⁡𝒩​\(𝒢joint,ϵ\)≲ϵ−\(ds\+da\)\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\lesssim\\epsilon^\{\-\\max\(d\_\{s\},d\_\{a\}\)\},\\qquad\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\epsilon\)\\lesssim\\epsilon^\{\-\(d\_\{s\}\+d\_\{a\}\)\}\.This is an upper\-bound comparison: under the same covering template, the factored entropy certificate depends onmax⁡\(ds,da\)\\max\(d\_\{s\},d\_\{a\}\)while the joint certificate depends onds\+dad\_\{s\}\+d\_\{a\}\. We discuss the absence of a matching lower bound in Remark[C\.7](https://arxiv.org/html/2607.13498#A3.SS7.SSS0.Px1)\.

### C\.5Rademacher Monotonicity Component of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\)

Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\) states that the inclusion𝒢fac⊆𝒢joint\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\mathrm\{joint\}\}propagates to both covering numbers and Rademacher complexities at every sample size\. The covering\-number half follows from monotonicity of𝒩​\(⋅,ϵ\)\\mathcal\{N\}\(\\cdot,\\epsilon\)under set inclusion \(Appendix[C\.3](https://arxiv.org/html/2607.13498#A3.SS3), Part \(ii\)\)\. The Rademacher half follows from a simpler argument than the covering bound: it requires only set inclusion, not entropy\.

###### Proposition\(Rademacher inequality from Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\)\)\.

Under the containment hypothesis of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\), for everynnand every i\.i\.d\. samplez1,…,znz\_\{1\},\\dots,z\_\{n\},

ℛn​\(𝒢fac\)≤ℛn​\(𝒢joint\)\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\)\\;\\leq\\;\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\)\.

###### Proof\.

The inclusion𝒢fac⊆𝒢joint\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\mathrm\{joint\}\}was established in Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\)\. For every fixed sampleSSand every sign realizationσ\\sigma,

supg∈𝒢fac\|1n​∑i=1nσi​g​\(zi\)\|≤supg∈𝒢joint\|1n​∑i=1nσi​g​\(zi\)\|,\\sup\_\{g\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}\\left\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sigma\_\{i\}\\,g\(z\_\{i\}\)\\right\|\\;\\leq\\;\\sup\_\{g\\in\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\}\\left\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sigma\_\{i\}\\,g\(z\_\{i\}\)\\right\|,because the supremum over a subset is no larger than the supremum over the superset\. Taking expectation overσ\\sigmagivesℛ^S​\(𝒢fac\)≤ℛ^S​\(𝒢joint\)\\widehat\{\\mathcal\{R\}\}\_\{S\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\)\\leq\\widehat\{\\mathcal\{R\}\}\_\{S\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\); taking expectation overSSgives the statedℛn\\mathcal\{R\}\_\{n\}inequality\. ∎

##### Remark \(Why not via Dudley?\)

An alternative would be to bound both Rademacher complexities via Dudley’s entropy integral and observe that𝒩​\(𝒢fac,ϵ\)≤𝒩​\(𝒢joint,ϵ\)\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\epsilon\)\\leq\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\epsilon\)propagates through the integral\. This argument only bounds the*upper bounds*on the two complexities and does not establish the ordering of the complexities themselves: the joint Rademacher complexity could in principle be much smaller than its Dudley upper bound\. Set inclusion gives the stronger statement\.

### C\.6Proof of Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2): Excess Risk for the SpectralL2L\_\{2\}Objective

We now address estimation\. The objective trained in practice is the noise\-contrastive estimator \(NCE\)\[Gutmann and Hyvärinen,[2010](https://arxiv.org/html/2607.13498#bib.bib49), Ma and Collins,[2018](https://arxiv.org/html/2607.13498#bib.bib29), Zhanget al\.,[2022](https://arxiv.org/html/2607.13498#bib.bib10), Gaoet al\.,[2025](https://arxiv.org/html/2607.13498#bib.bib12)\]for the transition\-ratio score, but the theoretical guarantees that bound representation error are derived in the spectralL2L\_\{2\}framework ofRenet al\.\[[2023](https://arxiv.org/html/2607.13498#bib.bib18)\]; we follow the same convention and analyze theL2L\_\{2\}ERM here\. The relationship between NCE optimization andL2L\_\{2\}population structure is discussed in Remark[C\.6](https://arxiv.org/html/2607.13498#A3.SS6.SSS0.Px2)and is*not*part of the formal claim of this proposition\.

Let

z=\(s,a,s′\)z=\(s,a,s^\{\\prime\}\)denote one observation, where

\(s,a\)∼ρ,s′∼P\(⋅∣s,a\)\.\(s,a\)\\sim\\rho,\\qquad s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\.For an i\.i\.d\. sample

zi=\(si,ai,si′\),i=1,…,n,z\_\{i\}=\(s\_\{i\},a\_\{i\},s^\{\\prime\}\_\{i\}\),\\qquad i=1,\\ldots,n,define the populationL2L\_\{2\}risk

L​\(g\):=𝔼ρ​\[∫𝒮\(P​\(s′∣s,a\)−g​\(s,a,s′\)\)2​𝑑ν​\(s′\)\]\.L\(g\):=\\mathbb\{E\}\_\{\\rho\}\\\!\\left\[\\int\_\{\\mathcal\{S\}\}\\bigl\(P\(s^\{\\prime\}\\mid s,a\)\-g\(s,a,s^\{\\prime\}\)\\bigr\)^\{2\}\\,d\\nu\(s^\{\\prime\}\)\\right\]\.\(19\)Also define

CP:=𝔼ρ​\[∫𝒮P​\(s′∣s,a\)2​𝑑ν​\(s′\)\],C\_\{P\}:=\\mathbb\{E\}\_\{\\rho\}\\\!\\left\[\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)^\{2\}\\,d\\nu\(s^\{\\prime\}\)\\right\],\(20\)and the centered population risk

L¯\(g\):=L\(g\)−CP\.\\bar\{L\}\(g\):=L\(g\)\-C\_\{P\}\.\(21\)The empirical objective is

L^n​\(g\):=1n​∑i=1nℓ​\(g;zi\),\\hat\{L\}\_\{n\}\(g\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(g;z\_\{i\}\),\(22\)where the per\-sample loss is

ℓ​\(g;z\):=−2​g​\(s,a,s′\)\+∫𝒮g2​\(s,a,u\)​𝑑ν​\(u\)\.\\ell\(g;z\):=\-2g\(s,a,s^\{\\prime\}\)\+\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,u\)\\,d\\nu\(u\)\.\(23\)We assumeCP<∞C\_\{P\}<\\inftythroughout\. This holds, for example, wheneverP\(⋅∣s,a\)∈L2\(ν\)P\(\\cdot\\mid s,a\)\\in L^\{2\}\(\\nu\)uniformly on the support ofρ\\rho\. In particular, under CP realizability with a uniformly bounded density representation\|P\(s′∣s,a\)\|≤Bg\|P\(s^\{\\prime\}\\mid s,a\)\|\\leq B\_\{g\}and withν​\(𝒮\)=1\\nu\(\\mathcal\{S\}\)=1, we have

CP=𝔼ρ​\[∫𝒮P​\(s′∣s,a\)2​𝑑ν​\(s′\)\]≤𝔼ρ​\[∫𝒮Bg2​𝑑ν​\(s′\)\]=Bg2\.C\_\{P\}=\\mathbb\{E\}\_\{\\rho\}\\\!\\left\[\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)^\{2\}\\,d\\nu\(s^\{\\prime\}\)\\right\]\\leq\\mathbb\{E\}\_\{\\rho\}\\\!\\left\[\\int\_\{\\mathcal\{S\}\}B\_\{g\}^\{2\}\\,d\\nu\(s^\{\\prime\}\)\\right\]=B\_\{g\}^\{2\}\.
We use the Rademacher\-complexity convention fixed in Appendix[C\.1](https://arxiv.org/html/2607.13498#A3.SS1)\. For the quadratic class below, the sample is the projected sample\(si,ai\)i=1n\(s\_\{i\},a\_\{i\}\)\_\{i=1\}^\{n\}\.

###### Lemma C\.2\(Covering transfer to the integrated quadratic class\)\.

Assumeν​\(𝒮\)=1\\nu\(\\mathcal\{S\}\)=1and\|g​\(s,a,s′\)\|≤Bg\|g\(s,a,s^\{\\prime\}\)\|\\leq B\_\{g\}for allg∈𝒢g\\in\\mathcal\{G\}and all\(s,a,s′\)\(s,a,s^\{\\prime\}\)\. Define

hg​\(s,a\):=∫𝒮g2​\(s,a,u\)​𝑑ν​\(u\),ℋ𝒢:=\{hg:g∈𝒢\}\.h\_\{g\}\(s,a\):=\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,u\)\\,d\\nu\(u\),\\qquad\\mathcal\{H\}\_\{\\mathcal\{G\}\}:=\\\{h\_\{g\}:g\\in\\mathcal\{G\}\\\}\.Then, for everyϵ\>0\\epsilon\>0,

𝒩​\(ℋ𝒢,ϵ\)≤𝒩​\(𝒢,ϵ2​Bg\)\.\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\},\\epsilon\)\\leq\\mathcal\{N\}\\\!\\left\(\\mathcal\{G\},\\frac\{\\epsilon\}\{2B\_\{g\}\}\\right\)\.\(24\)

###### Proof\.

If‖g−g′‖∞≤η\\\|g\-g^\{\\prime\}\\\|\_\{\\infty\}\\leq\\eta, then for every\(s,a\)\(s,a\),

\|hg​\(s,a\)−hg′​\(s,a\)\|\\displaystyle\|h\_\{g\}\(s,a\)\-h\_\{g^\{\\prime\}\}\(s,a\)\|=\|∫𝒮\(g2​\(s,a,u\)−g′⁣2​\(s,a,u\)\)​𝑑ν​\(u\)\|\\displaystyle=\\left\|\\int\_\{\\mathcal\{S\}\}\\bigl\(g^\{2\}\(s,a,u\)\-g^\{\\prime 2\}\(s,a,u\)\\bigr\)\\,d\\nu\(u\)\\right\|≤∫𝒮\|g​\(s,a,u\)−g′​\(s,a,u\)\|​\|g​\(s,a,u\)\+g′​\(s,a,u\)\|​𝑑ν​\(u\)\\displaystyle\\leq\\int\_\{\\mathcal\{S\}\}\|g\(s,a,u\)\-g^\{\\prime\}\(s,a,u\)\|\\,\|g\(s,a,u\)\+g^\{\\prime\}\(s,a,u\)\|\\,d\\nu\(u\)≤∫𝒮η⋅2​Bg​𝑑ν​\(u\)=2​Bg​η\.\\displaystyle\\leq\\int\_\{\\mathcal\{S\}\}\\eta\\cdot 2B\_\{g\}\\,d\\nu\(u\)=2B\_\{g\}\\eta\.Thus anyη\\eta\-cover of𝒢\\mathcal\{G\}in sup norm induces a2​Bg​η2B\_\{g\}\\eta\-cover ofℋ𝒢\\mathcal\{H\}\_\{\\mathcal\{G\}\}\. Settingη=ϵ/\(2​Bg\)\\eta=\\epsilon/\(2B\_\{g\}\)proves \([24](https://arxiv.org/html/2607.13498#A3.E24)\)\. ∎

###### Proposition\(Restatement of Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)\)\.

Let𝒢\\mathcal\{G\}be either𝒢fac\\mathcal\{G\}\_\{\\mathrm\{fac\}\}or𝒢joint\\mathcal\{G\}\_\{\\mathrm\{joint\}\}, and assume

sups,a,s′\|g​\(s,a,s′\)\|≤Bgfor all​g∈𝒢\.\\sup\_\{s,a,s^\{\\prime\}\}\|g\(s,a,s^\{\\prime\}\)\|\\leq B\_\{g\}\\qquad\\text\{for all \}g\\in\\mathcal\{G\}\.Assume also thatCP<∞C\_\{P\}<\\infty\. Let

g^n∈arg⁡ming∈𝒢⁡L^n​\(g\)\\hat\{g\}\_\{n\}\\in\\arg\\min\_\{g\\in\\mathcal\{G\}\}\\hat\{L\}\_\{n\}\(g\)and

g𝒢∗∈arg⁡ming∈𝒢⁡L​\(g\),g^\{\*\}\_\{\\mathcal\{G\}\}\\in\\arg\\min\_\{g\\in\\mathcal\{G\}\}L\(g\),both assumed to exist\. If the empirical minimizer exists only approximately, namely if

L^n​\(g^n\)≤infg∈𝒢L^n​\(g\)\+η,\\hat\{L\}\_\{n\}\(\\hat\{g\}\_\{n\}\)\\leq\\inf\_\{g\\in\\mathcal\{G\}\}\\hat\{L\}\_\{n\}\(g\)\+\\eta,then the same bound below holds with an additional\+η\+\\etaterm on the right\-hand side\.

Define the induced quadratic class

ℋ𝒢:=\{hg:𝒮×𝒜→ℝ:hg​\(s,a\)=∫𝒮g2​\(s,a,u\)​𝑑ν​\(u\),g∈𝒢\}\.\\mathcal\{H\}\_\{\\mathcal\{G\}\}:=\\left\\\{h\_\{g\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}\\;:\\;h\_\{g\}\(s,a\)=\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,u\)\\,d\\nu\(u\),\\quad g\\in\\mathcal\{G\}\\right\\\}\.Then, with probability at least1−δ1\-\\delta,

L​\(g^n\)≤L​\(g𝒢∗\)\+8​ℛn​\(𝒢\)\+4​ℛn​\(ℋ𝒢\)\+2​\(2​Bg\+Bg2\)​2​log⁡\(1/δ\)n\.L\(\\hat\{g\}\_\{n\}\)\\leq L\(g^\{\*\}\_\{\\mathcal\{G\}\}\)\+8\\,\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)\+4\\,\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)\+2\(2B\_\{g\}\+B\_\{g\}^\{2\}\)\\sqrt\{\\frac\{2\\log\(1/\\delta\)\}\{n\}\}\.\(25\)Hereℛn​\(ℋ𝒢\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)is computed on the projected sample\(si,ai\)i=1n\(s\_\{i\},a\_\{i\}\)\_\{i=1\}^\{n\}\.

Moreover, both Rademacher\-complexity terms admit Dudley upper bounds in terms of the covering number of𝒢\\mathcal\{G\}\. In particular,

𝒩​\(ℋ𝒢,ϵ\)≤𝒩​\(𝒢,ϵ2​Bg\)\.\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\},\\epsilon\)\\leq\\mathcal\{N\}\\\!\\left\(\\mathcal\{G\},\\frac\{\\epsilon\}\{2B\_\{g\}\}\\right\)\.\(26\)Thus the bound \([25](https://arxiv.org/html/2607.13498#A3.E25)\) can be expressed entirely in terms of the metric entropy of𝒢\\mathcal\{G\}\.

###### Proof\.

First, expanding the square inL​\(g\)L\(g\)gives

L​\(g\)\\displaystyle L\(g\)=𝔼ρ​∫𝒮\(P​\(s′∣s,a\)−g​\(s,a,s′\)\)2​𝑑ν​\(s′\)\\displaystyle=\\mathbb\{E\}\_\{\\rho\}\\\!\\int\_\{\\mathcal\{S\}\}\\bigl\(P\(s^\{\\prime\}\\mid s,a\)\-g\(s,a,s^\{\\prime\}\)\\bigr\)^\{2\}\\,d\\nu\(s^\{\\prime\}\)=CP−2​𝔼ρ​∫𝒮P​\(s′∣s,a\)​g​\(s,a,s′\)​𝑑ν​\(s′\)\+𝔼ρ​∫𝒮g2​\(s,a,s′\)​𝑑ν​\(s′\)\.\\displaystyle=C\_\{P\}\-2\\mathbb\{E\}\_\{\\rho\}\\\!\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)g\(s,a,s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\+\\mathbb\{E\}\_\{\\rho\}\\\!\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\.Sinces′∼P\(⋅∣s,a\)s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)conditional on\(s,a\)\(s,a\),

𝔼​\[g​\(s,a,s′\)\]=𝔼ρ​∫𝒮P​\(s′∣s,a\)​g​\(s,a,s′\)​𝑑ν​\(s′\)\.\\mathbb\{E\}\[g\(s,a,s^\{\\prime\}\)\]=\\mathbb\{E\}\_\{\\rho\}\\\!\\int\_\{\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)g\(s,a,s^\{\\prime\}\)\\,d\\nu\(s^\{\\prime\}\)\.Therefore

L¯​\(g\)=L​\(g\)−CP=𝔼​\[ℓ​\(g;z\)\]\.\\bar\{L\}\(g\)=L\(g\)\-C\_\{P\}=\\mathbb\{E\}\[\\ell\(g;z\)\]\.\(27\)In particular,L^n​\(g\)\\hat\{L\}\_\{n\}\(g\)is an unbiased empirical estimate of the centered riskL¯​\(g\)\\bar\{L\}\(g\), andLLandL¯\\bar\{L\}have the same minimizers because they differ only by the constantCPC\_\{P\}\. Using empirical optimality ofg^n\\hat\{g\}\_\{n\},

L​\(g^n\)−L​\(g𝒢∗\)\\displaystyle L\(\\hat\{g\}\_\{n\}\)\-L\(g^\{\*\}\_\{\\mathcal\{G\}\}\)=L¯​\(g^n\)−L¯​\(g𝒢∗\)\\displaystyle=\\bar\{L\}\(\\hat\{g\}\_\{n\}\)\-\\bar\{L\}\(g^\{\*\}\_\{\\mathcal\{G\}\}\)=\[L¯​\(g^n\)−L^n​\(g^n\)\]\+\[L^n​\(g^n\)−L^n​\(g𝒢∗\)\]\+\[L^n​\(g𝒢∗\)−L¯​\(g𝒢∗\)\]\\displaystyle=\[\\bar\{L\}\(\\hat\{g\}\_\{n\}\)\-\\hat\{L\}\_\{n\}\(\\hat\{g\}\_\{n\}\)\]\+\[\\hat\{L\}\_\{n\}\(\\hat\{g\}\_\{n\}\)\-\\hat\{L\}\_\{n\}\(g^\{\*\}\_\{\\mathcal\{G\}\}\)\]\+\[\\hat\{L\}\_\{n\}\(g^\{\*\}\_\{\\mathcal\{G\}\}\)\-\\bar\{L\}\(g^\{\*\}\_\{\\mathcal\{G\}\}\)\]≤2​supg∈𝒢\|L¯​\(g\)−L^n​\(g\)\|\.\\displaystyle\\leq 2\\sup\_\{g\\in\\mathcal\{G\}\}\|\\bar\{L\}\(g\)\-\\hat\{L\}\_\{n\}\(g\)\|\.\(28\)Ifg^n\\hat\{g\}\_\{n\}is only anη\\eta\-approximate empirical minimizer, the same argument adds an extra\+η\+\\etaterm\. Let

ℒ𝒢:=\{ℓg:z↦ℓ​\(g;z\):g∈𝒢\}\.\\mathcal\{L\}\_\{\\mathcal\{G\}\}:=\\\{\\ell\_\{g\}:z\\mapsto\\ell\(g;z\):g\\in\\mathcal\{G\}\\\}\.By the standard symmetrization inequality for the centered empirical process\[Bartlett and Mendelson,[2002](https://arxiv.org/html/2607.13498#bib.bib51), Wainwright,[2019](https://arxiv.org/html/2607.13498#bib.bib25)\],

𝔼​supg∈𝒢\|L¯​\(g\)−L^n​\(g\)\|≤2​ℛn​\(ℒ𝒢\)\.\\mathbb\{E\}\\sup\_\{g\\in\\mathcal\{G\}\}\|\\bar\{L\}\(g\)\-\\hat\{L\}\_\{n\}\(g\)\|\\leq 2\\mathcal\{R\}\_\{n\}\(\\mathcal\{L\}\_\{\\mathcal\{G\}\}\)\.\(29\)Now write

ℓ​\(g;z\)=−2​g​\(s,a,s′\)\+hg​\(s,a\),hg​\(s,a\):=∫𝒮g2​\(s,a,u\)​𝑑ν​\(u\)\.\\ell\(g;z\)=\-2g\(s,a,s^\{\\prime\}\)\+h\_\{g\}\(s,a\),\\qquad h\_\{g\}\(s,a\):=\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,u\)\\,d\\nu\(u\)\.By subadditivity and homogeneity of Rademacher complexity,

ℛn​\(ℒ𝒢\)≤2​ℛn​\(𝒢\)\+ℛn​\(ℋ𝒢\)\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{L\}\_\{\\mathcal\{G\}\}\)\\leq 2\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)\+\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)\.\(30\)Combining \([29](https://arxiv.org/html/2607.13498#A3.E29)\) and \([30](https://arxiv.org/html/2607.13498#A3.E30)\) gives

𝔼​supg∈𝒢\|L¯​\(g\)−L^n​\(g\)\|≤4​ℛn​\(𝒢\)\+2​ℛn​\(ℋ𝒢\)\.\\mathbb\{E\}\\sup\_\{g\\in\\mathcal\{G\}\}\|\\bar\{L\}\(g\)\-\\hat\{L\}\_\{n\}\(g\)\|\\leq 4\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)\+2\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)\.\(31\)The quadratic term is not controlled by the usual pointwise contraction principle, since

hg​\(s,a\)=∫𝒮g2​\(s,a,u\)​𝑑ν​\(u\)h\_\{g\}\(s,a\)=\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,u\)\\,d\\nu\(u\)integrates over all possible next statesuurather than applying a scalar function to the observed valueg​\(si,ai,si′\)g\(s\_\{i\},a\_\{i\},s^\{\\prime\}\_\{i\}\)\. Instead, Lemma[C\.2](https://arxiv.org/html/2607.13498#A3.Thmtheorem2)gives

𝒩​\(ℋ𝒢,ϵ\)≤𝒩​\(𝒢,ϵ2​Bg\),\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\},\\epsilon\)\\leq\\mathcal\{N\}\\\!\\left\(\\mathcal\{G\},\\frac\{\\epsilon\}\{2B\_\{g\}\}\\right\),which is the entropy estimate \([26](https://arxiv.org/html/2607.13498#A3.E26)\) stated in the proposition\. Dudley’s entropy integral\[Dudley,[1967](https://arxiv.org/html/2607.13498#bib.bib52), Wainwright,[2019](https://arxiv.org/html/2607.13498#bib.bib25)\]therefore bounds bothℛn​\(𝒢\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)andℛn​\(ℋ𝒢\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)in terms of the covering number of𝒢\\mathcal\{G\}, with the second bound using only the radius rescaling above\. It remains to pass from expectation to high probability\. Define

F​\(z1,…,zn\):=supg∈𝒢\|L¯​\(g\)−L^n​\(g\)\|\.F\(z\_\{1\},\\ldots,z\_\{n\}\):=\\sup\_\{g\\in\\mathcal\{G\}\}\|\\bar\{L\}\(g\)\-\\hat\{L\}\_\{n\}\(g\)\|\.Since\|g\|≤Bg\|g\|\\leq B\_\{g\}andν​\(𝒮\)=1\\nu\(\\mathcal\{S\}\)=1,

\|ℓ\(g;z\)\|≤2\|g\(s,a,s′\)\|\+∫𝒮g2\(s,a,u\)dν\(u\)≤2Bg\+Bg2=:Mg\.\|\\ell\(g;z\)\|\\leq 2\|g\(s,a,s^\{\\prime\}\)\|\+\\int\_\{\\mathcal\{S\}\}g^\{2\}\(s,a,u\)\\,d\\nu\(u\)\\leq 2B\_\{g\}\+B\_\{g\}^\{2\}=:M\_\{g\}\.Changing one sample point changesL^n​\(g\)\\hat\{L\}\_\{n\}\(g\)by at most2​Mg/n2M\_\{g\}/nuniformly overgg, and therefore changesFFby at most2​Mg/n2M\_\{g\}/n\. McDiarmid’s inequality\[McDiarmid,[1989](https://arxiv.org/html/2607.13498#bib.bib53)\]yields, with probability at least1−δ1\-\\delta,

F≤𝔼​F\+Mg​2​log⁡\(1/δ\)n\.F\\leq\\mathbb\{E\}F\+M\_\{g\}\\sqrt\{\\frac\{2\\log\(1/\\delta\)\}\{n\}\}\.Combining this display with \([31](https://arxiv.org/html/2607.13498#A3.E31)\) and then using \([28](https://arxiv.org/html/2607.13498#A3.E28)\), we obtain

L​\(g^n\)−L​\(g𝒢∗\)\\displaystyle L\(\\hat\{g\}\_\{n\}\)\-L\(g^\{\*\}\_\{\\mathcal\{G\}\}\)≤2​F\\displaystyle\\leq 2F≤8​ℛn​\(𝒢\)\+4​ℛn​\(ℋ𝒢\)\+2​Mg​2​log⁡\(1/δ\)n\.\\displaystyle\\leq 8\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)\+4\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)\+2M\_\{g\}\\sqrt\{\\frac\{2\\log\(1/\\delta\)\}\{n\}\}\.SubstitutingMg=2​Bg\+Bg2M\_\{g\}=2B\_\{g\}\+B\_\{g\}^\{2\}proves \([25](https://arxiv.org/html/2607.13498#A3.E25)\)\. ∎

##### Remark \(On the constants\)

The prefactors in \([25](https://arxiv.org/html/2607.13498#A3.E25)\) come directly from the proof\. The factor22in the excess\-risk decomposition \([28](https://arxiv.org/html/2607.13498#A3.E28)\), the factor22from symmetrization \([29](https://arxiv.org/html/2607.13498#A3.E29)\), and the decomposition

ℛn​\(ℒ𝒢\)≤2​ℛn​\(𝒢\)\+ℛn​\(ℋ𝒢\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{L\}\_\{\\mathcal\{G\}\}\)\\leq 2\\,\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)\+\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\)together give the coefficients88and44in front ofℛn​\(𝒢\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\)andℛn​\(ℋ𝒢\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\}\), respectively\. The tail coefficient comes from the uniform bound

\|ℓ​\(g;z\)\|≤2​Bg\+Bg2\|\\ell\(g;z\)\|\\leq 2B\_\{g\}\+B\_\{g\}^\{2\}and the bounded\-differences constant

2​\(2​Bg\+Bg2\)n\.\\frac\{2\(2B\_\{g\}\+B\_\{g\}^\{2\}\)\}\{n\}\.Thus these constants are explicit and problem\-dependent\. We avoid the phrase “universal constant” here because the dependence onBgB\_\{g\}is essential: it enters both through the per\-sample loss bound and through the covering\-number rescaling

𝒩​\(ℋ𝒢,ϵ\)≤𝒩​\(𝒢,ϵ2​Bg\)\.\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\},\\epsilon\)\\leq\\mathcal\{N\}\\\!\\left\(\\mathcal\{G\},\\frac\{\\epsilon\}\{2B\_\{g\}\}\\right\)\.

##### Remark \(Relationship to the NCE estimator\)

The ranking\-based NCE objective ofMa and Collins \[[2018](https://arxiv.org/html/2607.13498#bib.bib29)\]used by FaStR is consistent with maximum likelihood in the limit of infinitely many negative samples, but is not identical to theL2L\_\{2\}spectral ERM analyzed here\. Establishing a sharp finite\-sample equivalence between the two estimators is separate from the claim proved above\.

### C\.7Proof of Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3): Bound\-Implied Sufficient Sample Size

We combine the additive covering bound from Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)with the excess\-risk template from Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)\. The goal is to compare the sample sizes that this particular uniform\-convergence analysis*certifies*as sufficient for the factored class and for the joint class\.

Throughout this section, the word “certificate” means an upper bound produced by the displayed finite\-sample theorem\. Thus the comparison below is a comparison between two sufficient sample sizes obtained by inverting two upper bounds\. It is not a comparison between the true minimax sample complexities of the two statistical problems\. In particular, the joint\-class bound below is the standard uniform\-convergence certificate for an unrestricted Lipschitz state\-action encoder\. It is not a lower bound and may be loose for structured subclasses\.

###### Lemma C\.3\(Polynomial entropy implies a Rademacher rate\)\.

Letℱ\\mathcal\{F\}be a uniformly bounded real\-valued function class with

diam∞\(ℱ\):=supf,f′∈ℱ∥f−f′∥∞≤2R\.\\operatorname\{diam\}\_\{\\infty\}\(\\mathcal\{F\}\):=\\sup\_\{f,f^\{\\prime\}\\in\\mathcal\{F\}\}\\\|f\-f^\{\\prime\}\\\|\_\{\\infty\}\\leq 2R\.Assume that, for someA<∞A<\\inftyandD\>2D\>2,

log⁡𝒩​\(ℱ,τ\)≤A​τ−D,0<τ≤R\.\\log\\mathcal\{N\}\(\\mathcal\{F\},\\tau\)\\leq A\\tau^\{\-D\},\\qquad 0<\\tau\\leq R\.Then there exists a constantK​\(A,D,R\)<∞K\(A,D,R\)<\\infty, independent ofnn, such that

ℛn​\(ℱ\)≤K​\(A,D,R\)​n−1/D\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{F\}\)\\leq K\(A,D,R\)n^\{\-1/D\}\.\(32\)

###### Proof\.

Dudley’s entropy integral, together with the fact that a sup\-norm cover is also an empirical\-L2L\_\{2\}cover, gives for anyα∈\(0,R\]\\alpha\\in\(0,R\],

ℛn​\(ℱ\)≤4​α\+12n​∫αRlog⁡𝒩​\(ℱ,τ\)​𝑑τ\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{F\}\)\\leq 4\\alpha\+\\frac\{12\}\{\\sqrt\{n\}\}\\int\_\{\\alpha\}^\{R\}\\sqrt\{\\log\\mathcal\{N\}\(\\mathcal\{F\},\\tau\)\}\\,d\\tau\.Using the polynomial entropy bound,

ℛn​\(ℱ\)≤4​α\+12​An​∫αRτ−D/2​𝑑τ\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{F\}\)\\leq 4\\alpha\+\\frac\{12\\sqrt\{A\}\}\{\\sqrt\{n\}\}\\int\_\{\\alpha\}^\{R\}\\tau^\{\-D/2\}\\,d\\tau\.SinceD\>2D\>2,

∫αRτ−D/2​𝑑τ≤2D−2​α1−D/2\.\\int\_\{\\alpha\}^\{R\}\\tau^\{\-D/2\}\\,d\\tau\\leq\\frac\{2\}\{D\-2\}\\alpha^\{1\-D/2\}\.Chooseα=R​n−1/D\\alpha=Rn^\{\-1/D\}, which lies in\(0,R\]\(0,R\]forn≥1n\\geq 1\. Then

4​α=4​R​n−1/D,4\\alpha=4Rn^\{\-1/D\},and

1n​α1−D/2=1n​\(R​n−1/D\)1−D/2=R1−D/2​n−1/D\.\\frac\{1\}\{\\sqrt\{n\}\}\\alpha^\{1\-D/2\}=\\frac\{1\}\{\\sqrt\{n\}\}\(Rn^\{\-1/D\}\)^\{1\-D/2\}=R^\{1\-D/2\}n^\{\-1/D\}\.Therefore

ℛn​\(ℱ\)≤\(4​R\+24​AD−2​R1−D/2\)​n−1/D\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{F\}\)\\leq\\left\(4R\+\\frac\{24\\sqrt\{A\}\}\{D\-2\}R^\{1\-D/2\}\\right\)n^\{\-1/D\}\.The coefficient is finite and independent ofnn, proving the claim\. ∎

###### Proposition\(Restatement of Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)\)\.

Assume CP realizability, that is,ϵCP=0\\epsilon\_\{\\mathrm\{CP\}\}=0\. Also assume the containment condition of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\), so that the true CP transition model lies in both𝒢fac\\mathcal\{G\}\_\{\\mathrm\{fac\}\}and𝒢joint\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\.

Fixδ∈\(0,1\)\\delta\\in\(0,1\)\. Let

ℱs,ℱa,ℱm,ℱs​a\\mathcal\{F\}\_\{s\},\\qquad\\mathcal\{F\}\_\{a\},\\qquad\\mathcal\{F\}\_\{m\},\\qquad\\mathcal\{F\}\_\{sa\}beLL\-Lipschitz classes on compact domains\. Let

Dmax:=max⁡\(ds,da\),Dsum:=ds\+da,D\_\{\\max\}:=\\max\(d\_\{s\},d\_\{a\}\),\\qquad D\_\{\\mathrm\{sum\}\}:=d\_\{s\}\+d\_\{a\},and assume

Let

g^nfac∈arg⁡ming∈𝒢fac⁡L^n​\(g\),g^njoint∈arg⁡ming∈𝒢joint⁡L^n​\(g\)\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{fac\}\}\\in\\arg\\min\_\{g\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}\\hat\{L\}\_\{n\}\(g\),\\qquad\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{joint\}\}\\in\\arg\\min\_\{g\\in\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\}\\hat\{L\}\_\{n\}\(g\)be empirical minimizers for the two classes\.

Then, with probability at least1−δ1\-\\delta, the high\-probability bound of Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)yields the certificates

L​\(g^nfac\)\\displaystyle L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{fac\}\}\)≤Cfac​\(δ\)​n−1/Dmax,\\displaystyle\\leq C\_\{\\mathrm\{fac\}\}\(\\delta\)\\,n^\{\-1/D\_\{\\max\}\},L​\(g^njoint\)\\displaystyle L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{joint\}\}\)≤Cjoint​\(δ\)​n−1/Dsum,\\displaystyle\\leq C\_\{\\mathrm\{joint\}\}\(\\delta\)\\,n^\{\-1/D\_\{\\mathrm\{sum\}\}\},\(33\)where the constants

Cfac​\(δ\),Cjoint​\(δ\)C\_\{\\mathrm\{fac\}\}\(\\delta\),\\qquad C\_\{\\mathrm\{joint\}\}\(\\delta\)are finite, positive, independent ofnnand of the target accuracyϵ\\epsilon, and depend polynomially on the problem parameters

Bg,Bϕ,Bm,∞,L,diam⁡\(𝒮\),diam⁡\(𝒜\),d,B\_\{g\},\\quad B\_\{\\phi\},\\quad B\_\{m,\\infty\},\\quad L,\\quad\\operatorname\{diam\}\(\\mathcal\{S\}\),\\quad\\operatorname\{diam\}\(\\mathcal\{A\}\),\\quad d,and logarithmically on1/δ1/\\delta, with fixed dependence on the dimensionsds,dad\_\{s\},d\_\{a\}\.

Define the bound\-implied sufficient sample sizes

nfacsuff​\(ϵ\):=inf\{n∈ℕ:Cfac​\(δ\)​n−1/Dmax≤ϵ2\},n\_\{\\mathrm\{fac\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\):=\\inf\\left\\\{n\\in\\mathbb\{N\}:C\_\{\\mathrm\{fac\}\}\(\\delta\)\\,n^\{\-1/D\_\{\\max\}\}\\leq\\epsilon^\{2\}\\right\\\},and

njointsuff​\(ϵ\):=inf\{n∈ℕ:Cjoint​\(δ\)​n−1/Dsum≤ϵ2\}\.n\_\{\\mathrm\{joint\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\):=\\inf\\left\\\{n\\in\\mathbb\{N\}:C\_\{\\mathrm\{joint\}\}\(\\delta\)\\,n^\{\-1/D\_\{\\mathrm\{sum\}\}\}\\leq\\epsilon^\{2\}\\right\\\}\.These are the smallest sample sizes at which the certificates in \([33](https://arxiv.org/html/2607.13498#A3.E33)\) guarantee representation error at mostϵ2\\epsilon^\{2\}\. Then

njointsuff​\(ϵ\)nfacsuff​\(ϵ\)=Θ​\(ϵ−2​min⁡\(ds,da\)\)as​ϵ→0\.\\frac\{n\_\{\\mathrm\{joint\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}\{n\_\{\\mathrm\{fac\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}=\\Theta\\\!\\left\(\\epsilon^\{\-2\\min\(d\_\{s\},d\_\{a\}\)\}\\right\)\\qquad\\text\{as \}\\epsilon\\to 0\.\(34\)

###### Proof\.

Under CP realizability, there exists

gCP∈𝒢facg\_\{\\mathrm\{CP\}\}\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}such that

gCP​\(s,a,s′\)=P​\(s′∣s,a\)\.g\_\{\\mathrm\{CP\}\}\(s,a,s^\{\\prime\}\)=P\(s^\{\\prime\}\\mid s,a\)\.Hence

L​\(gCP\)=𝔼ρ​∫𝒮\(P​\(s′∣s,a\)−gCP​\(s,a,s′\)\)2​𝑑ν​\(s′\)=0\.L\(g\_\{\\mathrm\{CP\}\}\)=\\mathbb\{E\}\_\{\\rho\}\\\!\\int\_\{\\mathcal\{S\}\}\\bigl\(P\(s^\{\\prime\}\\mid s,a\)\-g\_\{\\mathrm\{CP\}\}\(s,a,s^\{\\prime\}\)\\bigr\)^\{2\}\\,d\\nu\(s^\{\\prime\}\)=0\.SinceL​\(g\)≥0L\(g\)\\geq 0for allgg, we have

L​\(g𝒢fac∗\)=0\.L\(g^\{\*\}\_\{\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}\)=0\.The containment condition gives

𝒢fac⊆𝒢joint,\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\mathrm\{joint\}\},so the samegCPg\_\{\\mathrm\{CP\}\}also belongs to𝒢joint\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\. Thus

L​\(g𝒢joint∗\)=0\.L\(g^\{\*\}\_\{\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\}\)=0\.Therefore the finite\-sample certificates contain only estimation and concentration terms\. By Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)and the Lipschitz entropy bound, there is a constantAfac<∞A\_\{\\mathrm\{fac\}\}<\\inftysuch that

log⁡𝒩​\(𝒢fac,τ\)≤Afac​τ−Dmax,Dmax:=max⁡\(ds,da\)\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\},\\tau\)\\leq A\_\{\\mathrm\{fac\}\}\\tau^\{\-D\_\{\\max\}\},\\qquad D\_\{\\max\}:=\\max\(d\_\{s\},d\_\{a\}\)\.For the unrestricted joint Lipschitz class, the standard product\-domain entropy bound gives

log⁡𝒩​\(𝒢joint,τ\)≤Ajoint​τ−Dsum,Dsum:=ds\+da\.\\log\\mathcal\{N\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\},\\tau\)\\leq A\_\{\\mathrm\{joint\}\}\\tau^\{\-D\_\{\\mathrm\{sum\}\}\},\\qquad D\_\{\\mathrm\{sum\}\}:=d\_\{s\}\+d\_\{a\}\.Moreover, by Lemma[C\.2](https://arxiv.org/html/2607.13498#A3.Thmtheorem2), the associated quadratic classes have the same entropy exponents up to constants:

log⁡𝒩​\(ℋ𝒢fac,τ\)≤Afac,H​τ−Dmax,\\log\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\},\\tau\)\\leq A\_\{\\mathrm\{fac\},H\}\\tau^\{\-D\_\{\\max\}\},and

log⁡𝒩​\(ℋ𝒢joint,τ\)≤Ajoint,H​τ−Dsum,\\log\\mathcal\{N\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\},\\tau\)\\leq A\_\{\\mathrm\{joint\},H\}\\tau^\{\-D\_\{\\mathrm\{sum\}\}\},for suitable finite constantsAfac,HA\_\{\\mathrm\{fac\},H\}andAjoint,HA\_\{\\mathrm\{joint\},H\}\. SinceDmax≥3D\_\{\\max\}\\geq 3and

Dsum=ds\+da≥Dmax,D\_\{\\mathrm\{sum\}\}=d\_\{s\}\+d\_\{a\}\\geq D\_\{\\max\},both entropy exponents are larger than22\. Applying Lemma[C\.3](https://arxiv.org/html/2607.13498#A3.Thmtheorem3)gives constantsKfac,Kjoint<∞K\_\{\\mathrm\{fac\}\},K\_\{\\mathrm\{joint\}\}<\\infty, independent ofnnandϵ\\epsilon, such that

ℛn​\(𝒢fac\)\+ℛn​\(ℋ𝒢fac\)≤Kfac​n−1/Dmax,\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\)\+\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}\)\\leq K\_\{\\mathrm\{fac\}\}n^\{\-1/D\_\{\\max\}\},and

ℛn​\(𝒢joint\)\+ℛn​\(ℋ𝒢joint\)≤Kjoint​n−1/Dsum\.\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\)\+\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\}\)\\leq K\_\{\\mathrm\{joint\}\}n^\{\-1/D\_\{\\mathrm\{sum\}\}\}\.Apply Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)to the two classes with confidence levelsδ/2\\delta/2and take a union bound\. With probability at least1−δ1\-\\delta, both bounds hold simultaneously\. Using the zero oracle risks above,

L​\(g^nfac\)≤Kfac′​n−1/Dmax\+2​\(2​Bg\+Bg2\)​2​log⁡\(2/δ\)n,L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{fac\}\}\)\\leq K\_\{\\mathrm\{fac\}\}^\{\\prime\}n^\{\-1/D\_\{\\max\}\}\+2\(2B\_\{g\}\+B\_\{g\}^\{2\}\)\\sqrt\{\\frac\{2\\log\(2/\\delta\)\}\{n\}\},and

L​\(g^njoint\)≤Kjoint′​n−1/Dsum\+2​\(2​Bg\+Bg2\)​2​log⁡\(2/δ\)n,L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{joint\}\}\)\\leq K\_\{\\mathrm\{joint\}\}^\{\\prime\}n^\{\-1/D\_\{\\mathrm\{sum\}\}\}\+2\(2B\_\{g\}\+B\_\{g\}^\{2\}\)\\sqrt\{\\frac\{2\\log\(2/\\delta\)\}\{n\}\},for finite constantsKfac′K\_\{\\mathrm\{fac\}\}^\{\\prime\}andKjoint′K\_\{\\mathrm\{joint\}\}^\{\\prime\}\. BecauseDmax≥3D\_\{\\max\}\\geq 3andDsum≥DmaxD\_\{\\mathrm\{sum\}\}\\geq D\_\{\\max\},

n−1/2≤n−1/Dmax,n−1/2≤n−1/Dsum,n≥1\.n^\{\-1/2\}\\leq n^\{\-1/D\_\{\\max\}\},\\qquad n^\{\-1/2\}\\leq n^\{\-1/D\_\{\\mathrm\{sum\}\}\},\\qquad n\\geq 1\.Absorbing the concentration terms into the constants yields

L​\(g^nfac\)\\displaystyle L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{fac\}\}\)≤Cfac​\(δ\)​n−1/Dmax,\\displaystyle\\leq C\_\{\\mathrm\{fac\}\}\(\\delta\)n^\{\-1/D\_\{\\max\}\},L​\(g^njoint\)\\displaystyle L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{joint\}\}\)≤Cjoint​\(δ\)​n−1/Dsum,\\displaystyle\\leq C\_\{\\mathrm\{joint\}\}\(\\delta\)n^\{\-1/D\_\{\\mathrm\{sum\}\}\},\(35\)which proves the certificates in \([33](https://arxiv.org/html/2607.13498#A3.E33)\)\. The constants are finite, positive, independent ofnnandϵ\\epsilon, and have the parameter dependence stated in the proposition\. It remains to invert the certificates\. The factored certificate guarantees

L​\(g^nfac\)≤ϵ2L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{fac\}\}\)\\leq\\epsilon^\{2\}whenever

Cfac​\(δ\)​n−1/Dmax≤ϵ2\.C\_\{\\mathrm\{fac\}\}\(\\delta\)n^\{\-1/D\_\{\\max\}\}\\leq\\epsilon^\{2\}\.This is equivalent to

n≥Cfac​\(δ\)Dmax​ϵ−2​Dmax\.n\\geq C\_\{\\mathrm\{fac\}\}\(\\delta\)^\{D\_\{\\max\}\}\\epsilon^\{\-2D\_\{\\max\}\}\.Similarly, the joint certificate guarantees

L​\(g^njoint\)≤ϵ2L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{joint\}\}\)\\leq\\epsilon^\{2\}whenever

n≥Cjoint​\(δ\)Dsum​ϵ−2​Dsum\.n\\geq C\_\{\\mathrm\{joint\}\}\(\\delta\)^\{D\_\{\\mathrm\{sum\}\}\}\\epsilon^\{\-2D\_\{\\mathrm\{sum\}\}\}\.If sample sizes are required to be integers, ceilings change these thresholds by a multiplicative1\+o​\(1\)1\+o\(1\)factor asϵ→0\\epsilon\\to 0\. Hence

njointsuff​\(ϵ\)nfacsuff​\(ϵ\)=Cjoint​\(δ\)DsumCfac​\(δ\)Dmax​ϵ−2​\(Dsum−Dmax\)​\(1\+o​\(1\)\)\.\\frac\{n\_\{\\mathrm\{joint\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}\{n\_\{\\mathrm\{fac\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}=\\frac\{C\_\{\\mathrm\{joint\}\}\(\\delta\)^\{D\_\{\\mathrm\{sum\}\}\}\}\{C\_\{\\mathrm\{fac\}\}\(\\delta\)^\{D\_\{\\max\}\}\}\\epsilon^\{\-2\(D\_\{\\mathrm\{sum\}\}\-D\_\{\\max\}\)\}\(1\+o\(1\)\)\.Finally,

Dsum−Dmax=ds\+da−max⁡\(ds,da\)=min⁡\(ds,da\)\.D\_\{\\mathrm\{sum\}\}\-D\_\{\\max\}=d\_\{s\}\+d\_\{a\}\-\\max\(d\_\{s\},d\_\{a\}\)=\\min\(d\_\{s\},d\_\{a\}\)\.Therefore

njointsuff​\(ϵ\)nfacsuff​\(ϵ\)=Θ​\(ϵ−2​min⁡\(ds,da\)\),\\frac\{n\_\{\\mathrm\{joint\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}\{n\_\{\\mathrm\{fac\}\}^\{\\mathrm\{suff\}\}\(\\epsilon\)\}=\\Theta\\\!\\left\(\\epsilon^\{\-2\\min\(d\_\{s\},d\_\{a\}\)\}\\right\),as claimed\. Since the thresholds were obtained by inverting upper\-bound certificates, this is an algebraic comparison of certified sufficient sample sizes, not a minimax lower bound\. ∎

##### Remark \(Scope and interpretation of the sample\-size comparison\)

Equation \([34](https://arxiv.org/html/2607.13498#A3.E34)\) compares the sample sizes that the same uniform\-convergence template certifies as sufficient for the two function classes\. The comparison is therefore between

Cfac​\(δ\)​n−1/DmaxC\_\{\\mathrm\{fac\}\}\(\\delta\)n^\{\-1/D\_\{\\max\}\}and

Cjoint​\(δ\)​n−1/Dsum,C\_\{\\mathrm\{joint\}\}\(\\delta\)n^\{\-1/D\_\{\\mathrm\{sum\}\}\},not between the true optimal risks achievable by all possible algorithms\.

In particular, \([34](https://arxiv.org/html/2607.13498#A3.E34)\) does*not*prove that learning the joint class has minimax sample complexity

Ω​\(ϵ−2​Dsum\)\.\\Omega\(\\epsilon^\{\-2D\_\{\\mathrm\{sum\}\}\}\)\.Such a statement would require a matching lower bound\. A matching lower bound would in turn require a packing construction showing that𝒢joint\\mathcal\{G\}\_\{\\mathrm\{joint\}\}contains many well\-separated functions at scaleϵ\\epsilon\. That kind of packing argument generally requires an additional richness assumption onℱs​a\\mathcal\{F\}\_\{sa\}, ensuring that the joint state\-action encoder can realize a sufficiently large family of distinct functions over the fullds\+dad\_\{s\}\+d\_\{a\}dimensional product domain\.

We do not impose such a richness assumption here\. Therefore, the conservative interpretation is the following: under the same generic uniform\-convergence analysis, the factored class receives a certificate with exponentDmax=max⁡\(ds,da\)D\_\{\\max\}=\\max\(d\_\{s\},d\_\{a\}\), whereas the unrestricted joint Lipschitz class receives the standard certificate with exponentDsum=ds\+daD\_\{\\mathrm\{sum\}\}=d\_\{s\}\+d\_\{a\}\. Inverting these two certificates yields a polynomially smaller bound\-implied sufficient sample size for the factored class\. Whether a more refined, algorithm\-specific analysis of the joint class can beat the generic uniform\-convergence certificate is a separate question\. The empirical results in the main paper are consistent with this conservative reading\.

##### Remark \(Non\-realizable case\)

WhenϵCP\>0\\epsilon\_\{\\mathrm\{CP\}\}\>0, the true transition model is not represented exactly by the CP\-factored class\. In that case, the factored class has an irreducible approximation term\. If

infg∈𝒢facL​\(g\)≤ϵCP2,\\inf\_\{g\\in\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}L\(g\)\\leq\\epsilon\_\{\\mathrm\{CP\}\}^\{2\},then the same finite\-sample argument gives the factored certificate

L​\(g^nfac\)≤ϵCP2\+Cfac​\(δ\)​n−1/Dmax\.L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{fac\}\}\)\\leq\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}\+C\_\{\\mathrm\{fac\}\}\(\\delta\)n^\{\-1/D\_\{\\max\}\}\.If the joint class still contains the true transition model, then its oracle risk remains zero, and its certificate is

L​\(g^njoint\)≤Cjoint​\(δ\)​n−1/Dsum\.L\(\\hat\{g\}\_\{n\}^\{\\,\\mathrm\{joint\}\}\)\\leq C\_\{\\mathrm\{joint\}\}\(\\delta\)n^\{\-1/D\_\{\\mathrm\{sum\}\}\}\.Thus the non\-realizable setting involves a bias–variance tradeoff\. The factored class has the smaller entropy exponent and hence the faster estimation term, but it may plateau at the approximation levelϵCP2\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}\. For moderate sample sizes and small CP approximation error, the factored certificate can still be sharper; for sufficiently large sample sizes, if the joint class is well\-specified and the factored class is not, the irreducible bias term may dominate the factored bound\. This is why the exact\-realizability result above is stated separately\.

##### Remark \(Same\-nncomparison without Lipschitz instantiation\)

The sample\-ratio in \([34](https://arxiv.org/html/2607.13498#A3.E34)\) relies on the Lipschitz covering bounds, which give the explicit dimensional exponentsDmaxD\_\{\\max\}andDsumD\_\{\\mathrm\{sum\}\}\. A weaker but more robust statement is available without any Lipschitz instantiation: under CP realizability and the containment condition of Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)\(ii\), at every fixednn, the certified upper bound onL2​\(ν\)L^\{2\}\(\\nu\)representation error from Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)is no larger for the factored class than for the joint class\.

The argument uses only set inclusion\. Appendix[C\.5](https://arxiv.org/html/2607.13498#A3.SS5)givesℛn​\(𝒢fac\)≤ℛn​\(𝒢joint\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\)\\leq\\mathcal\{R\}\_\{n\}\(\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\)from𝒢fac⊆𝒢joint\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\\subseteq\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\. The induced quadratic classℋ𝒢=\{\(s,a\)↦∫g2​\(s,a,u\)​𝑑ν​\(u\):g∈𝒢\}\\mathcal\{H\}\_\{\\mathcal\{G\}\}=\\\{\(s,a\)\\mapsto\\int g^\{2\}\(s,a,u\)\\,d\\nu\(u\):g\\in\\mathcal\{G\}\\\}inherits this monotonicity, sincehgh\_\{g\}is determined byggalone, soℛn​\(ℋ𝒢fac\)≤ℛn​\(ℋ𝒢joint\)\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\_\{\\mathrm\{fac\}\}\}\)\\leq\\mathcal\{R\}\_\{n\}\(\\mathcal\{H\}\_\{\\mathcal\{G\}\_\{\\mathrm\{joint\}\}\}\)\. Under CP realizability both approximation termsL​\(g𝒢∗\)L\(g^\{\*\}\_\{\\mathcal\{G\}\}\)in Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)equal zero; with the shared envelopeBgB\_\{g\}giving the same McDiarmid tail, the certificate \([25](https://arxiv.org/html/2607.13498#A3.E25)\) yields a no\-larger upper bound onL​\(g^n\)L\(\\hat\{g\}\_\{n\}\)for the factored class at the samenn\. This certifies an ordering of two upper bounds, not a minimax separation\.

### C\.8Conditional Implication for LSVI\-UCB

The representation results above control an average transition\-density error inL2​\(ν\)L^\{2\}\(\\nu\)\. This is the right object for spectral representation learning, but LSVI\-UCB requires a stronger condition: the Bellman backup must be uniformly approximable as a linear function of the feature used by the algorithm\. We state this downstream implication conditionally\. The point is not to derive this condition fromL2​\(ν\)L^\{2\}\(\\nu\)error alone, but to record what the factored structure would buy once the learned feature satisfies the standard approximate\-linear condition required by LSVI\-UCB\.

###### Definition 1\(\(η,𝒱,W\)\(\\eta,\\mathcal\{V\},W\)\-approximate Bellman linearity\)\.

A feature mapψ^:𝒮×𝒜→ℝd\\widehat\{\\psi\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}^\{d\}satisfies\(η,𝒱,W\)\(\\eta,\\mathcal\{V\},W\)\-approximate Bellman linearity if for everyV∈𝒱V\\in\\mathcal\{V\}there exists a vectorwV∈ℝdw\_\{V\}\\in\\mathbb\{R\}^\{d\}with‖wV‖2≤W\\\|w\_\{V\}\\\|\_\{2\}\\leq Wsuch that

sup\(s,a\)∈𝒮×𝒜\|r​\(s,a\)\+γ​𝔼s′∼P\(⋅∣s,a\)​\[V​\(s′\)\]−ψ^​\(s,a\)⊤​wV\|≤η\.\\sup\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\left\|r\(s,a\)\+\\gamma\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\[V\(s^\{\\prime\}\)\]\-\\widehat\{\\psi\}\(s,a\)^\{\\top\}w\_\{V\}\\right\|\\leq\\eta\.\(36\)Here𝒱\\mathcal\{V\}can be taken as the value\-function class generated by the LSVI\-UCB dynamic program\. A stronger but more conservative version is obtained by taking𝒱\\mathcal\{V\}to be all bounded functions with‖V‖∞≤Rmax/\(1−γ\)\\\|V\\\|\_\{\\infty\}\\leq R\_\{\\max\}/\(1\-\\gamma\)\.

###### Proposition C\.4\(Conditional LSVI\-UCB compatibility\)\.

Letψ^\\widehat\{\\psi\}be a learneddd\-dimensional feature map, withsups,a‖ψ^​\(s,a\)‖2≤Bψ\\sup\_\{s,a\}\\\|\\widehat\{\\psi\}\(s,a\)\\\|\_\{2\}\\leq B\_\{\\psi\}\. Suppose thatψ^\\widehat\{\\psi\}satisfies\(η,𝒱LSVI,W\)\(\\eta,\\mathcal\{V\}\_\{\\mathrm\{LSVI\}\},W\)\-approximate Bellman linearity in the sense of Definition[1](https://arxiv.org/html/2607.13498#Thmdefinition1)\. Assume that LSVI\-UCB is run withψ^\\widehat\{\\psi\}, the standard elliptical bonus, and the usual discounted\-to\-effective\-horizon reductionH=O~​\(\(1−γ\)−1\)H=\\widetilde\{O\}\(\(1\-\\gamma\)^\{\-1\}\)\.

If

η≤c0​\(1−γ\)​ϵ\\eta\\leq c\_\{0\}\(1\-\\gamma\)\\epsilonfor a sufficiently small numerical constantc0c\_\{0\}, then LSVI\-UCB returns anϵ\\epsilon\-optimal policy after

O~​\(poly​\(d,W,Bψ,\(1−γ\)−1,ϵ−1\)\)\\widetilde\{O\}\\\!\\left\(\\mathrm\{poly\}\\bigl\(d,W,B\_\{\\psi\},\(1\-\\gamma\)^\{\-1\},\\epsilon^\{\-1\}\\bigr\)\\right\)episodes\. Under the standard bounded\-norm normalization used in linear\-MDP analyses, this specializes to the familiar certificate

O~​\(d2\(1−γ\)4​ϵ2\)\.\\widetilde\{O\}\\\!\\left\(\\frac\{d^\{2\}\}\{\(1\-\\gamma\)^\{4\}\\epsilon^\{2\}\}\\right\)\.

###### Proof\.

Under approximate Bellman linearity, the Bellman target used by LSVI\-UCB is linear inψ^\\widehat\{\\psi\}up to a uniform misspecification errorη\\eta\. Standard misspecified\-linear\-MDP analyses for LSVI\-UCB then yield a regret decomposition of the form

Regret​\(T\)≤O~​\(d​T\(1−γ\)3/2\)\+Cmis​T​η1−γ,\\mathrm\{Regret\}\(T\)\\leq\\widetilde\{O\}\\\!\\left\(\\frac\{d\\sqrt\{T\}\}\{\(1\-\\gamma\)^\{3/2\}\}\\right\)\+C\_\{\\mathrm\{mis\}\}\\frac\{T\\eta\}\{1\-\\gamma\},\(37\)where the hidden factors depend polynomially on the feature and weight norm bounds\. The first term is the usual exact\-linear\-MDP regret term, and the second term is the cumulative Bellman misspecification error\.

To make the average regret at mostϵ\\epsilon, it suffices to chooseTTso that

d\(1−γ\)3/2​T≲ϵ,\\frac\{d\}\{\(1\-\\gamma\)^\{3/2\}\\sqrt\{T\}\}\\lesssim\\epsilon,which gives

T=O~​\(d2\(1−γ\)3​ϵ2\)\.T=\\widetilde\{O\}\\\!\\left\(\\frac\{d^\{2\}\}\{\(1\-\\gamma\)^\{3\}\\epsilon^\{2\}\}\\right\)\.At this value ofTT, the misspecification term in \([37](https://arxiv.org/html/2607.13498#A3.E37)\) contributes at most orderϵ​T\\epsilon Twhenever

η≲\(1−γ\)​ϵ\.\\eta\\lesssim\(1\-\\gamma\)\\epsilon\.Thus the conditionη≤c0​\(1−γ\)​ϵ\\eta\\leq c\_\{0\}\(1\-\\gamma\)\\epsilonensures that the misspecification term does not dominate the exact\-feature regret term\. Converting the average\-regret guarantee to a single output policy by the standard PAC conversion for discounted problems introduces one additional factor of\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}, yielding

O~​\(d2\(1−γ\)4​ϵ2\)\\widetilde\{O\}\\\!\\left\(\\frac\{d^\{2\}\}\{\(1\-\\gamma\)^\{4\}\\epsilon^\{2\}\}\\right\)under the usual bounded\-norm normalization\. Keeping the norm constants explicit gives the stated polynomial dependence ond,W,Bψ,\(1−γ\)−1d,W,B\_\{\\psi\},\(1\-\\gamma\)^\{\-1\}andϵ−1\\epsilon^\{\-1\}\. ∎

##### Implication for the factored representation\.

Under exact CP realizability, the true feature

ψ∗​\(s,a\)=ϕs∗​\(s\)⊙ϕa∗​\(a\)\\psi^\{\*\}\(s,a\)=\\phi\_\{s\}^\{\*\}\(s\)\\odot\\phi\_\{a\}^\{\*\}\(a\)satisfies Bellman linearity exactly: for every boundedVV,

r​\(s,a\)\+γ​𝔼​\[V​\(s′\)∣s,a\]=ψ∗​\(s,a\)⊤​wVr\(s,a\)\+\\gamma\\mathbb\{E\}\[V\(s^\{\\prime\}\)\\mid s,a\]=\\psi^\{\*\}\(s,a\)^\{\\top\}w\_\{V\}for an appropriate coefficient vectorwVw\_\{V\}\. Thus the CP feature is a valid linear\-MDP feature\. The learned featureψ^\\widehat\{\\psi\}can be used by LSVI\-UCB once it satisfies the stronger uniform condition \([36](https://arxiv.org/html/2607.13498#A3.E36)\)\. The representation\-learning results in Appendix[C\.6](https://arxiv.org/html/2607.13498#A3.SS6)do not by themselves prove this uniform condition from averageL2​\(ν\)L^\{2\}\(\\nu\)error; rather, they provide a route for reducing the number of representation\-learning samples needed to obtain an accurate transition feature\.

The potential advantage of the factored structure is therefore conditional but clear\. If the factored and joint learned features are both evaluated through the same downstream LSVI\-UCB analysis, then the online exploration guarantee depends on the feature dimension and the Bellman misspecification levelη\\eta\. The factored representation can improve the upstream sample requirement for reaching a smallη\\etabecause its representation class has a smaller certified estimation term in the low\-CP\-bias regime\. Once the thresholdη≤c0​\(1−γ\)​ϵ\\eta\\leq c\_\{0\}\(1\-\\gamma\)\\epsilonis reached, the downstream LSVI\-UCB guarantee is the standard one\.

## Appendix DAblation Study and Analysis

This appendix gathers two complementary analyses of the FaStR design\. Section[D\.1](https://arxiv.org/html/2607.13498#A4.SS1)is a controlled encoder ablation that isolates which architectural ingredient drives the empirical gain by interpolating between CTRL\-SR and FaStR through two intermediate architectures\. Section[D\.2](https://arxiv.org/html/2607.13498#A4.SS2)is an offline diagnostic that estimates the cost of imposing the CP factorization on the dynamics of each task, independently of the RL training pipeline\. The two analyses answer distinct questions:*which property of the encoder produces the gain*, and*when the underlying CP assumption is a good fit for the environment*\.

### D\.1Interaction Structure of the Spectral Encoder

FaStR differs from CTRL\-SR in three design choices: the encoder is split into separate state and action networks, their outputs are combined through element\-wise multiplication, and the NCE logit is a diagonal trilinear transition\-ratio score rather than a fused state\-action logit\. We isolate the contribution of each choice with a four\-condition ablation in which two intermediate architectures \(concat\-of\-separate and Tucker\) sit between the monolithic baseline and FaStR\.

#### D\.1\.1Encoder Structure of the Four Conditions

The four conditions share the TD3 backbone, optimization, RP\-NCE protocol, and evaluation setup of Appendix[B](https://arxiv.org/html/2607.13498#A2)\(Sections[B\.1](https://arxiv.org/html/2607.13498#A2.SS1)and[B\.2](https://arxiv.org/html/2607.13498#A2.SS2)\); they differ*only*in the encoder used to produce the spectral feature and, where the encoder structure forces it, in the matching critic\.

##### Condition \(a\): CTRL\-SR\.

The baseline ofGaoet al\.\[[2025](https://arxiv.org/html/2607.13498#bib.bib12)\], identical to Appendix[B\.3](https://arxiv.org/html/2607.13498#A2.SS3): a monolithic encoderφ​\(s,a\)∈ℝ512\\varphi\(s,a\)\\in\\mathbb\{R\}^\{512\}on\[s;a\]\[s;a\], bilinear NCE logitφ​\(s,a\)⊤​μ​\(s′\)\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\), and an RFF\-MLP critic onφ\\varphi\. Effective NCE dimensiond=512d\{=\}512\.

##### Condition \(b\): Concat\-of\-separate\.

Two independent networksϕs:𝒮→ℝ256\\phi\_\{s\}\{:\}\\,\\mathcal\{S\}\\to\\mathbb\{R\}^\{256\}andϕa:𝒜→ℝ256\\phi\_\{a\}\{:\}\\,\\mathcal\{A\}\\to\\mathbb\{R\}^\{256\}, concatenated and projected through a learned linear layer with Mish activation:ψ=σ​\(W​\[ϕs;ϕa\]\+b\)\\psi=\\sigma\(W\[\\phi\_\{s\};\\,\\phi\_\{a\}\]\+b\),W∈ℝ512×512W\\in\\mathbb\{R\}^\{512\\times 512\}\. The 256\-dim outputs are sized so that the joint feature matches the 512\-dimψ\\psiof the other conditions\. Each component ofψ\\psidepends additively onϕs\\phi\_\{s\}andϕa\\phi\_\{a\}\. The critic is the same RFF\-MLP onψ\\psias in FaStR\. Effective NCE dimensiond=512d\{=\}512\.

##### Condition \(c\): Tucker\.

Two independent networksϕs:𝒮→ℝ64\\phi\_\{s\}\{:\}\\,\\mathcal\{S\}\\to\\mathbb\{R\}^\{64\}andϕa:𝒜→ℝ64\\phi\_\{a\}\{:\}\\,\\mathcal\{A\}\\to\\mathbb\{R\}^\{64\}feeding a full trilinear score:h​\(s,a,s′\)=ϕs​\(s\)⊤​M​\(s′\)​ϕa​\(a\)h\(s,a,s^\{\\prime\}\)=\\phi\_\{s\}\(s\)^\{\\top\}M\(s^\{\\prime\}\)\\,\\phi\_\{a\}\(a\)withM​\(s′\)∈ℝ64×64M\(s^\{\\prime\}\)\\in\\mathbb\{R\}^\{64\\times 64\}\. This is the Tucker parameterization of Section[3\.1](https://arxiv.org/html/2607.13498#S3.SS1), with effective NCE dimensionds⋅da=4096d\_\{s\}\\cdot d\_\{a\}=4096\. Because full trilinear scoring is incompatible with the Hadamard critic, we use a structurally matched critic: a bilinear termϕs⊤​W​ϕa\\phi\_\{s\}^\{\\top\}W\\phi\_\{a\}plus an RFF\-MLP residual on\[ϕs;ϕa\]\[\\phi\_\{s\};\\phi\_\{a\}\]\. The dimensionds=da=64d\_\{s\}\{=\}d\_\{a\}\{=\}64keeps total parameters and per\-step floating\-point operations \(FLOPs\) within 10% of FaStR’s\.

##### Condition \(d\): FaStR \(ours\)\.

Identical to Appendix[B\.2](https://arxiv.org/html/2607.13498#A2.SS2): diagonal trilinear NCE score\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\)withϕs,ϕa,m→ℝ512\\phi\_\{s\},\\phi\_\{a\},m\\to\\mathbb\{R\}^\{512\}, factored critic featureψ=ϕs​\(s\)⊙ϕa​\(a\)\\psi=\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\), and an RFF\-MLP critic onψ\\psi\. Effective NCE dimensiond=512=Bd\{=\}512\{=\}B\.

The critic architecture varies only where the interaction structure forces it: FaStR’s diagonal trilinear score induces the same Hadamard feature used by the critic, while Tucker’s full interaction matrix requires a bilinear\-residual head\.

#### D\.1\.2Results

Figure[4](https://arxiv.org/html/2607.13498#A4.F4)reports learning curves on five DM Control Suite tasks at two action dimensions\. Humanoid tasks \(dim𝒜=21\\dim\\mathcal\{A\}\{=\}21\) are the primary comparison because the factored advantage is largest in high\-dimensional action spaces; quadruped tasks \(dim𝒜=12\\dim\\mathcal\{A\}\{=\}12\) act as a control where all methods are expected to perform comparably\.

![Refer to caption](https://arxiv.org/html/2607.13498v1/x5.png)Figure 4:NCE transition\-ratio score structure ablation\. Learning curves on five DM Control Suite tasks at two action dimensions: Humanoid \(dim𝒜=21\\dim\\mathcal\{A\}\{=\}21\) is the primary comparison; Quadruped \(dim𝒜=12\\dim\\mathcal\{A\}\{=\}12\) is a control\. Curves: mean return across seeds; shaded:±1\\pm 1std\.##### Encoder separation alone does not explain the gain\.

The \(a\)\-vs\-\(b\) contrast isolates the effect of using separateϕs\\phi\_\{s\}andϕa\\phi\_\{a\}networks without multiplicative interaction\. Concat preserves the split but combines through a learned affine transformation, so each component ofψ\\psiis a linear mixture ofϕs\\phi\_\{s\}andϕa\\phi\_\{a\}\. On humanoid tasks, concat gives a modest improvement over the monolithic encoder; on quadruped, the two are within noise\. The improvement is well below FaStR’s gain on the same tasks, ruling out the hypothesis that the split architecture alone accounts for FaStR’s advantage\.

##### The diagonal trilinear interaction is the source of the gain\.

The \(b\)\-vs\-\(d\) contrast isolates the CP interaction in the transition\-ratio score versus linear projection with encoder separation held constant\. FaStR gives higher returns than concat on every humanoid task, despite using the same 512\-dimensionalψ\\psiand the same RFF\-MLP critic\. The structural reason is that concat gives an additive scoring functionψk=∑j\(Ws\)k​j​ϕs,j\+∑j\(Wa\)k​j​ϕa,j\+bk\\psi\_\{k\}=\\sum\_\{j\}\(W\_\{s\}\)\_\{kj\}\\phi\_\{s,j\}\+\\sum\_\{j\}\(W\_\{a\}\)\_\{kj\}\\phi\_\{a,j\}\+b\_\{k\}, mixing all dimensions, while FaStR’s NCE logit contains the per\-component productϕs,k​\(s\)⋅ϕa,k​\(a\)⋅mk​\(s′\)\\phi\_\{s,k\}\(s\)\\cdot\\phi\_\{a,k\}\(a\)\\cdot m\_\{k\}\(s^\{\\prime\}\): thekk\-th action feature directly scales thekk\-th state feature before comparison with thekk\-th next\-state factor\. This per\-component product is the algebraic structure required by Theorem[3\.1](https://arxiv.org/html/2607.13498#S3.Thmtheorem1)for the log covering number to decompose additively overℱs\\mathcal\{F\}\_\{s\},ℱa\\mathcal\{F\}\_\{a\},ℱm\\mathcal\{F\}\_\{m\}\. Concat loses this decomposition at the projection layer, whereWWintroduces cross\-mode coupling\.

##### Tucker is more expressive but trains worse under NCE\.

The \(c\)\-vs\-\(d\) contrast tests whether the strictly more expressive Tucker trilinear score translates into better contrastive performance\. Tucker can represent arbitrary cross\-component productsϕs,i⋅ϕa,j\\phi\_\{s,i\}\\cdot\\phi\_\{a,j\},i≠ji\\neq j, that diagonal CP excludes\. Empirically Tucker gives lower returns than FaStR on humanoid tasks and matches FaStR on quadruped, where the gradient\-coverage constraint is weaker\.

The cause is the gradient rank deficiency of Section[3\.1](https://arxiv.org/html/2607.13498#S3.SS1):∇vec⁡M​\(sj′\)ℓ\\nabla\_\{\\operatorname\{vec\}M\(s^\{\\prime\}\_\{j\}\)\}\\elllies inspan​\{ϕa​\(ai\)⊗ϕs​\(si\)\}i=1B\\mathrm\{span\}\\\{\\phi\_\{a\}\(a\_\{i\}\)\\otimes\\phi\_\{s\}\(s\_\{i\}\)\\\}\_\{i=1\}^\{B\}, a subspace of dimension at mostmin⁡\(B,ds​da\)\\min\(B,d\_\{s\}d\_\{a\}\)\. In condition \(c\),ds​da=4096d\_\{s\}d\_\{a\}\{=\}4096whileB=512B\{=\}512, so each mini\-batch constrains only512512of40964096parameter directions ofM​\(s′\)M\(s^\{\\prime\}\)\. Three requirements then conflict: encoder capacity, NCE validity \(B≥ds​daB\\geq d\_\{s\}d\_\{a\}\), and gradient coverage in a single batch\. At standard batch size, Tucker cannot satisfy all three simultaneously\.

##### Low\-rank Tucker does not recover the gap\.

The natural fix is to lowerds,dad\_\{s\},d\_\{a\}untilds​da≤Bd\_\{s\}d\_\{a\}\\leq B\. The boundary caseds=da=22d\_\{s\}\{=\}d\_\{a\}\{=\}22\(ds​da=484≤Bd\_\{s\}d\_\{a\}\{=\}484\\leq B\) on humanoid\-walk gives near\-zero return: encoder dimension becomes the binding constraint\. Any low\-rank Tucker variant within the validity constraint lies between this point and dense Tucker, so neither end matches CP\.

FaStR avoids the trilemma by restrictingM​\(s′\)M\(s^\{\\prime\}\)to a diagonalm​\(s′\)∈ℝdm\(s^\{\\prime\}\)\\in\\mathbb\{R\}^\{d\}, withddparameters per next\-state and gradient spanning all ofℝd\\mathbb\{R\}^\{d\}wheneverB≥dB\\geq d\(d=B=512d\{=\}B\{=\}512here\)\. The rank\-one chain\-rule structure of Section[3\.1](https://arxiv.org/html/2607.13498#S3.SS1)holds for any trilinear\-score lossℓ=f​\(ϕs⊤​M​ϕa\)\\ell=f\(\\phi\_\{s\}^\{\\top\}M\\phi\_\{a\}\), so the same mechanism applies to the energy\-based score matching of SPEDER\[Renet al\.,[2023](https://arxiv.org/html/2607.13498#bib.bib18)\]and the diffusion objective of Diff\-SR\[Shribaket al\.,[2024](https://arxiv.org/html/2607.13498#bib.bib21)\]\.

##### What CP gives up\.

CP excludes cross\-component productsϕs,i⋅ϕa,j\\phi\_\{s,i\}\\cdot\\phi\_\{a,j\}withi≠ji\\neq j, which introduces approximation bias when action dimensions are tightly coupled\. Whenever the Tucker core is approximately superdiagonal after a change of basis, the deep encoders can absorb the rotation internally and the CP restriction does not reduce expressiveness\. CP’s statistical advantage can also outweigh the approximation bias under limited training data, which is the typical online RL regime\. Tucker is preferable when the training objective is not constrained by contrastive trilinear scoring \(e\.g\., regression or MLE with enough data to spanℝds​da\\mathbb\{R\}^\{d\_\{s\}d\_\{a\}\}\), as in prior bilinear\-MDP work\[Duet al\.,[2021](https://arxiv.org/html/2607.13498#bib.bib37), Yang and Wang,[2020](https://arxiv.org/html/2607.13498#bib.bib26)\], and when the environment has cross\-component couplings that no encoder rotation can absorb\.

### D\.2Empirical CP Misfit Diagnostic

The finite\-sample certificate of Proposition[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)decomposes representation error into a CP approximation biasϵCP2\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}and an estimation term whose rate depends on the covering number\. Proposition[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)shows that, under low CP bias, the estimation advantage of the factored class over the joint class scales withmin⁡\(ds,da\)\\min\(d\_\{s\},d\_\{a\}\)\. The single\-task experiments are consistent with this prediction but do not measureϵCP\\epsilon\_\{\\mathrm\{CP\}\}directly\. We construct an offline diagnostic that estimates the transition\-modeling cost of enforcing CP, independently of the RL training pipeline, providing a structural complement to the per\-task return numbers in Table[1](https://arxiv.org/html/2607.13498#S4.T1)\.

#### D\.2\.1Implementation Details

For each of the 13 DM Control Suite tasks in Figure[5](https://arxiv.org/html/2607.13498#A4.F5), we collect a replay buffer of 500k transitions from a trained CTRL\-SR agent across 5 seeds\. Buffers are sourced from CTRL\-SR rather than FaStR so that the state\-action distribution is not shaped by the CP\-factored encoder under evaluation, and they cover a range from early random exploration to trained locomotion, reflecting the RL\-relevant state distribution\.

We train two transition\-modeling predictors on each buffer: the CP predictor scores transitions asgCP​\(s,a,s′\)=\(ϕs​\(s\)⊙ϕa​\(a\)\)⊤​m​\(s′\)g\_\{\\mathrm\{CP\}\}\(s,a,s^\{\\prime\}\)=\(\\phi\_\{s\}\(s\)\\odot\\phi\_\{a\}\(a\)\)^\{\\top\}m\(s^\{\\prime\}\)with separate state and action encoders, and the joint predictor usesgjoint​\(s,a,s′\)=φ​\(s,a\)⊤​μ​\(s′\)g\_\{\\mathrm\{joint\}\}\(s,a,s^\{\\prime\}\)=\\varphi\(s,a\)^\{\\top\}\\mu\(s^\{\\prime\}\)with a single monolithic encoder\. Both share feature dimension \(d=512d\{=\}512\), encoder depth \(2 residual blocks with LayerNorm and Mish\), learning rate \(10−410^\{\-4\}\), batch size \(512\), training epochs \(50\), and noise schedule \(25 VP levels\)\. The CP predictor has more parameters in every task, making the comparison conservative: if it still incurs higher test loss, the gap reflects structural misfit rather than underfitting\. Each predictor is trained with the RP\-NCE ranking loss plus an auxiliary conditional\-moment head \(weight 0\.1\) that predicts standardized next\-state features \(raw observations and random Fourier projections\)\.

Data is split 70/15/15 into train, validation, and test\. We select the checkpoint with the lowest validation NCE loss and report two test\-set metrics: the NCE misfitΔ^CPNCE=ℒCPtest−ℒjointtest\\widehat\{\\Delta\}\_\{\\mathrm\{CP\}\}^\{\\mathrm\{NCE\}\}=\\mathcal\{L\}\_\{\\mathrm\{CP\}\}^\{\\mathrm\{test\}\}\-\\mathcal\{L\}\_\{\\mathrm\{joint\}\}^\{\\mathrm\{test\}\}, which matches the contrastive objective used in the RL pipeline, and the moment misfitΔ^CPmom\\widehat\{\\Delta\}\_\{\\mathrm\{CP\}\}^\{\\mathrm\{mom\}\}, which is independent of NCE\. Both are distribution\-weighted empirical proxies forϵCP2\\epsilon\_\{\\mathrm\{CP\}\}^\{2\}rather than direct estimates\.

#### D\.2\.2Results

Figure[5](https://arxiv.org/html/2607.13498#A4.F5)reports the diagnostic across 13 tasks with action dimensions from 2 to 38\. The CP factorization incurs a small excess transition\-modeling loss in nearly all tasks\. Walker\-Walk is an outlier on both metrics: it has the highest NCE misfit and the highest moment misfit of any task in the figure\.

![Refer to caption](https://arxiv.org/html/2607.13498v1/x6.png)Figure 5:Empirical CP misfit diagnostic across 13 DM Control Suite tasks\. Each marker is one task; thexx\-axis is the test\-set excess loss of the CP predictor over the joint predictor \(positive means CP is worse\), measured by \(a\) NCE misfitΔ^CPNCE\\widehat\{\\Delta\}\_\{\\mathrm\{CP\}\}^\{\\mathrm\{NCE\}\}and \(b\) moment misfitΔ^CPmom\\widehat\{\\Delta\}\_\{\\mathrm\{CP\}\}^\{\\mathrm\{mom\}\}\(log scale\); theyy\-axis is the final\-return gain FaStR−\-CTRL\-SR from Figure[2](https://arxiv.org/html/2607.13498#S4.F2)\. Marker color and size encode action dimensiondim𝒜\\dim\\mathcal\{A\}\. The CP predictor has more parameters than the joint predictor in every task\.##### Two\-condition pattern\.

The gains match the prediction of Propositions[3\.2](https://arxiv.org/html/2607.13498#S3.Thmtheorem2)and[3\.3](https://arxiv.org/html/2607.13498#S3.Thmtheorem3)\. Among low\-dimensional tasks \(dim𝒜≤6\\dim\\mathcal\{A\}\\leq 6, lower\-left cluster of Figure[5](https://arxiv.org/html/2607.13498#A4.F5)\), CP fits the dynamics well, but the covering\-number separation at smallmin⁡\(ds,da\)\\min\(d\_\{s\},d\_\{a\}\)is too small to yield a measurable return difference\. Among high\-dimensional tasks \(dim𝒜≥12\\dim\\mathcal\{A\}\\geq 12\), where the theory predicts a larger estimation advantage, gains appear in every case where CP misfit is low: the Humanoid and Dog points sit in the upper region of both panels, while the high\-misfit Walker\-Walk point, with the same action dimension as Cheetah\-Run, has no gain, which corresponds to the regime of Remark[C\.7](https://arxiv.org/html/2607.13498#A3.SS7.SSS0.Px2)in which the bias term offsets the estimation advantage\. Neither low misfit nor high action dimensionality alone is sufficient: both conditions must hold together\.

##### Scope of the diagnostic\.

The misfit metrics do not rank\-order gain magnitudes within the low\-misfit group\. This is expected, since the estimation advantage depends both on the action dimension and on the effective complexity of the dynamics within the factored class, neither of which is captured by a scalar proxy\. We therefore use the diagnostic as a compatibility check for the CP assumption rather than as a continuous predictor of gain magnitude\.