Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL

# Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL
Source: [https://arxiv.org/html/2607.00392](https://arxiv.org/html/2607.00392)
###### Abstract

Unsupervised Reinforcement Learning \(URL\) aims to pre\-train scalable, skill\-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks\. Despite recent progress, we argue that current off\-policy URL methods are limited by two critical, overlooked bottlenecks: \(1\) non\-stationary skill semantics and \(2\) brittle generalization\. To address these challenges, we propose GenDa \(Generalizable Data\-efficient Agent\), a unified framework for robust unsupervised reinforcement learning\. First, we introduce a skill relabeling mechanism to mitigate non\-stationarity and significantly improve data efficiency for pre\-training\. Second, we propose a Complementary Information Bottleneck \(CIB\), encouraging the learned skill policy to focus on ego\-centric features and become robust to distribution shifts for downstream tasks\. Through various experiments, we demonstrate that GenDa significantly enhances the scalability of URL with superior generalizability and data efficiency\. Our code and videos are available at[https://ihatebroccoli\.github\.io/official\-GenDa/](https://ihatebroccoli.github.io/official-GenDa/)\.

Machine Learning, ICML

## 1Introduction

Recent advances in unsupervised reinforcement learning \(URL\) have enabled the discovery of semantically distinct behaviors \(“skills”\) from state transitions, without access to external reward signals\(Gregoret al\.,[2016](https://arxiv.org/html/2607.00392#bib.bib5); Kimet al\.,[2021](https://arxiv.org/html/2607.00392#bib.bib6); Kamiennyet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib7); Parket al\.,[2023](https://arxiv.org/html/2607.00392#bib.bib8); Yanget al\.,[2023](https://arxiv.org/html/2607.00392#bib.bib9)\)\. This paradigm aims to create a general\-purpose foundation for control rather than just learning individual skills\. It provides a pre\-trained policy that readily adapts to diverse downstream tasks with minimal fine\-tuning\(Laskinet al\.,[2021](https://arxiv.org/html/2607.00392#bib.bib10); Rajeswaret al\.,[2023](https://arxiv.org/html/2607.00392#bib.bib11)\)\. Achieving scalability in URL requires both efficiency during pre\-training and robustness during transfer, ensuring the learned policy generalizes to varied downstream tasks\.

Despite recent progress in URL, we argue that current state\-of\-the\-art methods face two critical bottlenecks that hinder this scalability: \(1\) sample inefficiency arising from the overlooked non\-stationary skill semantics, and \(2\) brittle generalization caused by overfitting to global context\.

First, real\-world interactions are costly, making data efficiency important\. To maximize data efficiency, modern URL methods rely on off\-policy algorithms that reuse past experiences stored in a replay buffer\. However, this introduces a fatal flaw:semantic drift\. In off\-policy settings, each trajectory is collected under a randomly sampled skillzzandzz\-conditioned skill policy\(Eysenbachet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib1); Sharmaet al\.,[2020](https://arxiv.org/html/2607.00392#bib.bib13)\)\. The resultingzz–trajectory pair is stored in a replay buffer and reused throughout training\(Haarnojaet al\.,[2018](https://arxiv.org/html/2607.00392#bib.bib14); Laskinet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib2); Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3); Zhenget al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib4)\)\. As learning progresses, the behavioral trajectories induced by the same skillzzcan change, because skill policy evolves across off\-policy learning\. The time\-varying semantics ofzzinduce destabilization of the skill policy by providing stale samples\.

Second, for a skill policy to be a scalable foundation for various downstream tasks, it must be robust to distribution shifts\. A skill such as “walking forward” should be executable regardless of the global contextual information, such as environmental factors\. However, in many prior works\(Eysenbachet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib1); Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3); Zhenget al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib4)\), the observations used for training skill policies include global contextual information that is not essential for executing skills\. When skill policies are conditioned on these global signals, the same skill may exhibit drastically different behaviors under slight shifts in coordinate distributions or become overfitted to specific global contexts\. This brittleness limits the reusability of skills in downstream tasks and leads to fundamental generalization failures\.

To address these challenges:

- •We formalize two overlooked failure modes and provide theoretical and empirical evidence of their impact\.
- •We introduce skill relabeling to mitigate the semantic drift in the previous off\-policy URL\. We also propose a novel regularizer to ensure skill diversity under the mutual information objective, effectively differentiating our skill relabeling framework from prior literature in other fields\(Andrychowiczet al\.,[2017](https://arxiv.org/html/2607.00392#bib.bib36)\)\.
- •We propose a Complementary Information Bottleneck \(CIB\), a drop\-in module compatible with an unsupervised framework\. CIB learns an embedding that prevents the policy from exploiting global contextual information, improving skill execution consistency under distribution shifts\.

We evaluate our approach on a diverse set of state and pixel benchmarks, demonstrating superior data efficiency and skill\-policy generalization compared to prior methods\. Notably, we also consider high\-dimensional state environments, where existing approaches fail to discover meaningful skills, while our method successfully learns coherent and reusable skill policies\.

## 2Preliminaries

##### Unsupervised skill discovery

We follow the settings of prior work\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\), a controlled Markov process without a reward function\. MDP is defined asℳ=\(𝒮,𝒜,μ,p\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mu,p\)\.𝒮\\mathcal\{S\}represents the state space,𝒜\\mathcal\{A\}represents the action space,μ​:​Δ​\(𝒮\)\{\\mu\\text\{: \}\\Delta\(\\mathcal\{S\}\)\}represents the initial state distribution, andp​:​𝒮×𝒜→Δ​\(𝒮\)p\\text\{: \}\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\Delta\(\\mathcal\{S\}\)represents the transition dynamics kernel\. We also consider a discrete or continuous set of latent vectorsz∈𝒵z\\in\\mathcal\{Z\}with a latent\-conditioned skill policyπ​\(a∣s,z\)\\pi\(a\\mid s,z\)\.

In the terminology of unsupervised skill discovery, the latent skill vectorzzis sampled from the prior distributionp​\(z\)p\(z\)\. Then skill policyπ​\(a∣s,z\)\\pi\(a\\mid s,z\)rolls out a trajectoryτ=\(s0,a0,s1,a1,…,sT\)\\tau=\(s\_\{0\},a\_\{0\},s\_\{1\},a\_\{1\},\.\.\.,s\_\{T\}\)with a fixedzzfor the entire episode\. In this setting, the main goal is to learn diverse and useful behaviorsπ​\(a∣s,z\)\\pi\(a\\mid s,z\)from scratch without guidance \(e\.g\., data, prior knowledge, and supervision\)\.

We follow the representation objective of the metric\-aware unsupervised skill discovery\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\)with latent mapping functionϕ​\(s\)\\phi\(s\)to inject semantic meaning in the skill vectorz∈𝒵z\\in\\mathcal\{Z\}:

supπ,ϕ𝔼p​\(τ,z\)​\[∑t=0T−1\(ϕ​\(s′\)−ϕ​\(s\)\)⊤​z\]s\.t\.​‖ϕ​\(s\)−ϕ​\(s′\)‖2≤1,∀\(s,s′\)∈𝒮a​d​j\\begin\{split\}\\sup\_\{\\pi,\\phi\}\\mathbb\{E\}\_\{p\}\(\\tau,z\)\\bigg\[\\sum^\{T\-1\}\_\{t=0\}\(\\phi\(s^\{\\prime\}\)\-\\phi\(s\)\)^\{\\top\}z\\bigg\]\\\\ \\text\{s\.t\.\}\\\|\\phi\(s\)\-\\phi\(s^\{\\prime\}\)\\\|\_\{2\}\\leq 1,\\forall\(s,s^\{\\prime\}\)\\in\\mathcal\{S\}\_\{adj\}\\end\{split\}\(1\)

##### Downstream application with pre\-trained skill policy

A pre\-trained skill policyπ\\pican be adapted to downstream tasks\. A hierarchical controllerπh​\(z\|s\)\\pi^\{h\}\(z\|s\)samples from the set of learned skills and deliverszzto the \(frozen\) skill policy to maximize the downstream reward\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x1.png)Figure 1:Semantic drift in prior algorithm\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\)\.This figure shows trajectories collected by the skill policy using the same skill vector z, illustrating that different behavioral trajectories can be observed for the same z during training\. In standard off\-policy URL, these trajectories are all stored in the replay buffer and repeatedly reused for representation learning\. As a result, a one\-to\-many mapping between z and trajectories emerges, which destabilizes representation learning and leads to semantic drift\.

## 3Problem Statements

We formally identify the two core challenges preventing off\-policy URL from achieving scalability\.

### 3\.1Sample Inefficiency from Semantically Ungrounded Skill Rollouts

In the off\-policy URL setting, the agent collects a trajectoryτ\\tauconditioned on a sampled skillz∼p​\(z\)z\\sim p\(z\)\. The transition tuple, including the skill label\(τ,zr​o​l​l\)\(\\tau,z\_\{roll\}\), is stored in a replay bufferℬ\\mathcal\{B\}\. The critical issue lies in the dual role ofzz: it serves as both the input skill for the policyπ​\(a\|s,z\)\\pi\(a\|s,z\)and the target label for the representationϕ​\(s\)\\phi\(s\)\.

Unlike standard RL, where the reward function is static, in URL, the semantic meaning ofzzis defined by the representationϕ\\phi, which is continuously evolving\. Letϕt\\phi\_\{t\}denote the representation at training steptt\. A trajectoryτ\\taugenerated at timeto​l​dt\_\{old\}with skillzzsatisfied the relationshipz≈ϕto​l​d​\(τ\)z\\approx\\phi\_\{t\_\{old\}\}\(\\tau\)\. However, at the current steptc​u​r​rt\_\{curr\}, the updated representationϕtc​u​r​r\\phi\_\{t\_\{curr\}\}may map this trajectory to a completely different point in the latent space\.

Most existing methods ignore this drift, updating the current policy using the stale pair\(τ,z\)\(\\tau,z\)from the buffer\. This creates a semantic misalignment: the policy is penalized for failing to generatezzaccording to the currentϕtc​u​r​r\\phi\_\{t\_\{curr\}\}, even though the trajectory was correct underϕto​l​d\\phi\_\{t\_\{old\}\}\. Consequently, no meaningful correction occurs until the agent gathers enough “fresh” samples to effectively displace or outweigh the stale data\. This acts as a source of high\-variance noise in the gradient estimation\. Consequently, increasing the sample reuse rate \(i\.e\., high UTD ratio\)—which typically improves efficiency in off\-policy RL—ironically worsens instability in URL by overfitting to this persistent label noise\.

Figure[1](https://arxiv.org/html/2607.00392#S2.F1)illustrates that the same skill z can be associated with different trajectories depending on the training stage ofϕ\\phiandπ\\pi, highlighting a potentially critical failure mode in the off\-policy URL framework\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x2.png)Figure 2:Overfitting to xy\-coordinates \(global context\)\.An offset \(a,b\) indicates that the agent’s initial xy position is set to \(a,b\) at evaluation time\. During training, the agent always starts from \(0,0\), so this setup evaluates whether a given skill z produces consistent behavior under shifted initial conditions\. Curves of the same color represent trajectories generated by the same skill z\. When different offsets are applied, the skill policy\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\)fails to reproduce consistent trajectories, providing clear evidence of brittle generalization\.
### 3\.2Lack of Skill\-Policy Generalization under Downstream Tasks

A second limitation concerns the limited generalization capability of the learned skill policy\. In URL, skill policies are typically intended to function as consistent local behavioral primitives that respond to local state changes, directed by the given skill’s guidance\. This property is a key prerequisite for reusing skill policy across diverse initial conditions and downstream environments\.

In most prior works, however, skill policies are often conditioned directly on the full state observation\. Such observations include global contextual information \(e\.g\., absolute positions or visual background cues\) that is not essential for executing local behaviors\. When the policy is conditioned on this information, skill execution becomes entangled with the global context, weakening the intended locality of the learned behaviors\.

This entanglement makes the policy fragile under distribution shifts\. Changes in the global context between pre\-training and deployment cause the same latentzzto produce unreliable behaviors, significantly hindering downstream performance\.

In Figure[2](https://arxiv.org/html/2607.00392#S3.F2), we show a failure mode of the prior algorithm where conditioning on full state observation fails to execute reliably on varying global contexts\. This failure highlights that maximizing state coverage during pre\-training is a necessary but insufficient condition for scalability; the learned skills must also be disentangled from the specific global context in which they were discovered\.

## 4Method

We propose GenDa \(Generalizable Data\-efficient Agent\), a unified framework designed to train a scalable skill foundation for control\. Unlike prior works that treat skill discovery as a static optimization problem, GenDa addresses the dynamic nature of off\-policy learning and structural requirements for generalization\. We introduce \(a\) skill relabeling to dynamically align past experiences with the evolving latent space, and \(b\) Complementary Information Bottleneck \(CIB\) to structurally encourage the policy to learn ego\-centric, reusable behaviors\. Our approach is based on the previous state\-of\-the\-art algorithm\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\)\.

### 4\.1Relabeling for Skill Representation Learning

#### 4\.1\.1Relabeling with the current latent interpretation\.

Standard off\-policy URL methods suffer fromsemantic drift, where the skill labelszzstored in the replay buffer become stale as the representationϕ\\phievolves\. Updating the policy with these stale labels injects high\-variance noise into the learning process, thereby limiting the data efficiency in the off\-policy setting\. To overcome this, we introduce skill relabeling, which treats past trajectories not as fixed \(τ,z\\tau,z\) pairs, but as flexible experiences that can be re\-interpreted\.

##### Relabeling \(zz\-step\)

To correct the semantic inconsistency caused by fixed rollout\-timezr​o​l​lz\_\{roll\}in the replay buffer, we need to define a new labelzzthat is consistent with the currentϕ\\phi\. To define thatzz, we revisit the telescoping sum in Equation[1](https://arxiv.org/html/2607.00392#S2.E1)\. For an episodeτ=\(s0,a0,s1,…,sT\)\\tau=\(s\_\{0\},a\_\{0\},s\_\{1\},\.\.\.,s\_\{T\}\), the telescoping sum can be written as\(ϕ​\(sT\)−ϕ​\(s0\)\)⊤​z\(\\phi\(s\_\{T\}\)\-\\phi\(s\_\{0\}\)\)^\{\\top\}z, which means∑t=0T−1\(ϕ​\(st\+1\)−ϕ​\(st\)\)\\sum\\limits\_\{t=0\}^\{T\-1\}\{\(\\phi\(s\_\{t\+1\}\)\-\\phi\(s\_\{t\}\)\)\}\. We can define this telescoping sum as delivering the most appropriate direction vector to explainτ\\tauwith on\-policyϕ\\phi\.

Instead of usingzr​o​l​lz\_\{roll\}, we computezrelabz\_\{\\text\{relab\}\}from the current latent function:

zrelab​\(τ\)=unit​\(ϕ​\(sT\)−ϕ​\(s0\)\),unit​\(x\):=xmax​\(‖x‖2,ϵ\)\.\\begin\{split\}z\_\{\\text\{relab\}\}\(\\tau\)=\\mathrm\{unit\}\\\!\\left\(\\phi\(s\_\{T\}\)\-\\phi\(s\_\{0\}\)\\right\),\\\\ \\mathrm\{unit\}\(x\):=\\frac\{x\}\{\\text\{max\}\(\\\|x\\\|\_\{2\},\\epsilon\)\}\.\\end\{split\}\(2\)whereϵ\\epsilonis a very small positive scalar\.

##### Optimization \(ϕ\\phi\-step\)

We then learn with the objective in equation[1](https://arxiv.org/html/2607.00392#S2.E1)by replacingzzwithzrelabz\_\{\\text\{relab\}\}while stopping gradients:

supπ,ϕ𝔼p​\(τ,zrelab\)​\[∑t=0T−1\(ϕ​\(s′\)−ϕ​\(s\)\)⊤​zrelab\]s\.t\.​‖ϕ​\(s\)−ϕ​\(s′\)‖2≤1,∀\(s,s′\)∈𝒮a​d​j\\begin\{split\}\\sup\_\{\\pi,\\phi\}\\mathbb\{E\}\_\{p\}\(\\tau,z\_\{\\text\{relab\}\}\)\\bigg\[\\sum^\{T\-1\}\_\{t=0\}\(\\phi\(s^\{\\prime\}\)\-\\phi\(s\)\)^\{\\top\}z\_\{\\text\{relab\}\}\\bigg\]\\\\ \\text\{s\.t\.\}\\\|\\phi\(s\)\-\\phi\(s^\{\\prime\}\)\\\|\_\{2\}\\leq 1,\\forall\(s,s^\{\\prime\}\)\\in\\mathcal\{S\}\_\{adj\}\\end\{split\}\(3\)
Iterating \(*zz\-step*→\\rightarrow*ϕ\\phi\-step*\) mitigates the gradient conflicts caused by the fixedzrollz\_\{\\text\{roll\}\}, thereby resolving the aforementioned semantic drift\. We demonstrate that our iterative update is mathematically robust and structurally stable at each step \(Details in Appendix[A](https://arxiv.org/html/2607.00392#A1)\)\.

zr​e​l​a​b=unit​\(ϕtgt​\(sT\)−ϕtgt​\(s0\)\)\.\\displaystyle z\_\{relab\}=\\mathrm\{unit\}\\\!\\left\(\\phi\_\{\\text\{tgt\}\}\(s\_\{T\}\)\-\\phi\_\{\\text\{tgt\}\}\(s\_\{0\}\)\\right\)\.\(4\)

##### Implementation details of skill relabeling

In practice, to reduce training instability, we compute the relabeled targets using an exponential moving average \(EMA\) network\(Schwarzeret al\.,[2021](https://arxiv.org/html/2607.00392#bib.bib35)\), as shown in Equation[4](https://arxiv.org/html/2607.00392#S4.E4)\.

#### 4\.1\.2Preventing collapse via a uniformity regularizer\.

In prior works utilizing a fixedzr​o​l​lz\_\{roll\}label, the distribution ofzr​o​l​lz\_\{roll\}is naturally guaranteed to follow a random distributionp​\(z\)p\(z\), which is fixed after sampling\. Consequently, the entropy termH​\(z\)H\(z\)becomes a trivial constant in the objective forϕ\\phi\.

While relabeling stabilizes learning, naively maximizing alignment with re\-assigned labels can lead torepresentation collapse, whereϕ\\phimaps all states to a narrow region of the latent space to trivially satisfy the objective \(Equation[3](https://arxiv.org/html/2607.00392#S4.E3)\)\. Since the distribution of the relabeled skillzr​e​l​a​bz\_\{relab\}is not guaranteed to followp​\(z\)p\(z\), to prevent such collapse, we introduce a contrastive\-style*uniformity*regularizer to explicitly maximizeH​\(zr​e​l​a​b\)H\(z\_\{relab\}\)\.

Letvi=unit​\(ϕ​\(sT\(i\)\)−ϕ​\(s0\(i\)\)\)v\_\{i\}=\\mathrm\{unit\}\\\!\\left\(\\phi\(s\_\{T\}^\{\(i\)\}\)\-\\phi\(s\_\{0\}^\{\(i\)\}\)\\right\)be the episodic direction for episodeiiin a mini\-batch\. We define

ℒunif=1B​∑i=1Blog​∑j=1j≠iBexp⁡\(vi⊤​vj\),\\displaystyle\\mathcal\{L\}\_\{\\text\{unif\}\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\sum\_\{\\begin\{subarray\}\{c\}j=1\\\\ j\\neq i\\end\{subarray\}\}^\{B\}\\exp\\\!\\left\(v\_\{i\}^\{\\top\}v\_\{j\}\\right\),\(5\)whereBBis the number of samples in a mini\-batch\. We*minimize*ℒunif\\mathcal\{L\}\_\{\\text\{unif\}\}, which improves the lower bound ofH​\(zr​e​l​a​b\)H\(z\_\{relab\}\)\(van den Oordet al\.,[2018](https://arxiv.org/html/2607.00392#bib.bib25)\)\. By encouraging the skill vectors to be uniformly distributed on the hypersphere, we ensure that the agent discovers a diverse repertoire of behaviors even while continuously redefining what those behaviors are\.

#### 4\.1\.3Final representation objective\.

Letℒbase\\mathcal\{L\}\_\{\\text\{base\}\}denote the representation loss for the objective in Equation[3](https://arxiv.org/html/2607.00392#S4.E3)\. Our final representation learning loss is

ℒϕ=ℒbase\+β​ℒunif\\displaystyle\\mathcal\{L\}\_\{\\phi\}=\\mathcal\{L\}\_\{\\text\{base\}\}\+\\beta\\,\\mathcal\{L\}\_\{\\text\{unif\}\}\(6\)whereβ\\betais a scalar coefficient \(we useβ=1\.0\\beta=1\.0for all environments\)\. Figure[9](https://arxiv.org/html/2607.00392#A2.F9)\(in Appendix\) depicts that our uniformity term can handle the induced bias well\. In Appendix[B\.1](https://arxiv.org/html/2607.00392#A2.SS1), we provide a theoretical analysis demonstrating that relabeling stabilizes the representation learning process in an off\-policy setting\.

### 4\.2Relabeling for Skill Policy Learning

In the policy learning phase, the intrinsic reward function

r​\(ϕ,z\)=\(ϕ​\(s′\)−ϕ​\(s\)\)⊤​z\\displaystyle r\(\\phi,z\)=\(\\phi\(s^\{\\prime\}\)\-\\phi\(s\)\)^\{\\top\}z\(7\)evaluates how well a transitions→s′s\\rightarrow s^\{\\prime\}follows a conditionedzzin the latent space based on the magnitude and direction ofΔ​ϕ\\Delta\\phi\. Prior methods use only fixedzrollz\_\{\\text\{roll\}\}as a condition of the intrinsic reward function, which limits the privilege of the off\-policy setting\.

We instead relabel each transition with multiple targets derived from diverse state pairs within the same trajectory to enrich the off\-policy reward signal by exploiting the intrinsic reward in Equation[7](https://arxiv.org/html/2607.00392#S4.E7)\.

For a transition at timett, we sample a horizonc\>0c\>0and define:

zc=unit​\(ϕ​\(st\+c\)−ϕ​\(st\)\)\.\\displaystyle z\_\{c\}=\\mathrm\{unit\}\\\!\\left\(\\phi\(s\_\{t\+c\}\)\-\\phi\(s\_\{t\}\)\\right\)\.\(8\)We mixzcz\_\{c\}with thezrelabz\_\{\\text\{relab\}\}and the original rollout labelzrollz\_\{\\text\{roll\}\}\(details in Appendix[C](https://arxiv.org/html/2607.00392#A3)\) as commonly done in relabeling to preserve on\-policy semantics in practice\(Andrychowiczet al\.,[2017](https://arxiv.org/html/2607.00392#bib.bib36)\)\. This enables the policy/value function to learn from a richer set of\(s,s′,z\)\(s,s^\{\\prime\},z\)configurations than those directly experienced, improving bootstrapping\.

Algorithm 1GenDa Unsupervised Skill Discovery1:Initialize skill policy

π​\(a∣ℓ,z\)\\pi\(a\\mid\\ell,z\), representation function

ϕ\\phi, EMA of representation function

ϕt​g​t\\phi\_\{tgt\}, CIB

qψ​\(ℓ∣s\)q\_\{\\psi\}\(\\ell\\mid s\), Lagrange multiplier

λ\\lambda, replay buffer

ℬ\\mathcal\{B\}
2:for

i←1i\\leftarrow 1to \(\# epochs\)do

3:for

j←1j\\leftarrow 1to \(\# episodes per epoch\)do

4:Sample skill

z∼p​\(z\)z\\sim p\(z\)
5:Collect trajectory

τ\\tauusing

π​\(a∣ℓ,z\)\\pi\(a\\mid\\ell,z\)with

ℓ∼qψ\(⋅∣s\)\\ell\\sim q\_\{\\psi\}\(\\cdot\\mid s\), and store to

ℬ\\mathcal\{B\}
6:endfor

7:for

k←1k\\leftarrow 1to \(\# gradient steps per epoch\)do

8:Sample a batch

x=\(st,at,st\+1,st\+c,s0,sT,zr​o​l​l\)​from​ℬx=\(s\_\{t\},a\_\{t\},s\_\{t\+1\},s\_\{t\+c\},s\_\{0\},s\_\{T\},z\_\{roll\}\)\\text\{ from \}\\mathcal\{B\}
9:

xr​e​l​a​b←x\_\{relab\}\\leftarrowRelabel

zr​o​l​lz\_\{roll\}in

xxto

zr​e​l​a​bz\_\{relab\}using Eq\.[4](https://arxiv.org/html/2607.00392#S4.E4)

10:

xc←x\_\{c\}\\leftarrowRelabel

zr​o​l​lz\_\{roll\}in

xxto

zcz\_\{c\}using Eq\.[8](https://arxiv.org/html/2607.00392#S4.E8)

11:Update representation

ϕ​\(s\)\\phi\(s\)and Lagrange multiplier

λ\\lambdausing

xr​e​l​a​bx\_\{relab\}with Eq\.[6](https://arxiv.org/html/2607.00392#S4.E6)

12:Create mixed batch:

xm​i​x←x∪xr​e​l​a​b∪xcx\_\{mix\}\\leftarrow x\\cup x\_\{relab\}\\cup x\_\{c\}
13:Update skill policy

π​\(a∣ℓ,z\)\\pi\(a\\mid\\ell,z\)using SAC\(Haarnojaet al\.,[2018](https://arxiv.org/html/2607.00392#bib.bib14)\)with

xm​i​xx\_\{mix\}
14:Update CIB to maximize Eq\.[9](https://arxiv.org/html/2607.00392#S4.E9)

15:Update EMA target:

ϕt​g​t←α​ϕt​g​t\+\(1−α\)​ϕ\\phi\_\{tgt\}\\leftarrow\\alpha\\phi\_\{tgt\}\+\(1\-\\alpha\)\\phi
16:endfor

17:endfor

### 4\.3Complementary Information Bottleneck \(CIB\)

A pre\-trained foundational skill policyπ​\(a∣s,z\)\\pi\(a\\mid s,z\)is desired to execute pre\-trained skills for diverse applications, where distributional shifts in global contextual information frequently occur\. However, previous algorithms use the full state as an observation that leads to unintended overfitting on the global context\. To disentangle this connection, we propose the Complementary Information Bottleneck \(CIB\)\.

#### 4\.3\.1Key idea\.

Metric\-aware skill discovery methods encourage the latent spaceϕ\\phito capture temporal distances\. Often, the global contextual information is desired for theϕ\\philatent space, because this objective promotes the agent to capture the extensive “temporal” manifolds in the state space\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\)\. We desire to make the policy rely on the skill without entangling global contextual information in the raw state\. To address this, we propose substituting the raw policy observations with a learned embedding that is*complementary*toϕ​\(s\)\\phi\(s\)\.

We train an encoderqψ​\(ℓ∣s\)q\_\{\\psi\}\(\\ell\\mid s\)that produces an embeddingℓ\\ell, together with a decoderpω​\(s^∣ℓ,ϕ​\(s\)\)p\_\{\\omega\}\(\\hat\{s\}\\mid\\ell,\\phi\(s\)\), by maximizing a variational information bottleneck\(Alemiet al\.,[2017](https://arxiv.org/html/2607.00392#bib.bib37)\)style objective:

𝒥CIB=I​\(ℓ;s\)−I​\(ℓ;ϕ​\(s\)\)\\displaystyle\\mathcal\{J\}\_\{\\text\{CIB\}\}=I\(\\ell;s\)\-I\(\\ell;\\phi\(s\)\)\(9\)
Intuitively, the decoder can useϕ​\(s\)\\phi\(s\)to reconstruct whatever informationϕ\\phialready captures; thus,qψ​\(ℓ∣s\)q\_\{\\psi\}\(\\ell\\mid s\)is encouraged to preserve the remaining information needed to reconstructss, makingℓ\\ellcomplementary toϕ​\(s\)\\phi\(s\)\.

#### 4\.3\.2Skill Policy with CIB\.

We replace the raw state input of the skill policy with the CIB embeddingℓ\\ell:

π​\(a∣s,z\)⟶π​\(a∣ℓ,z\),ℓ∼qψ​\(ℓ∣s\)\.\\displaystyle\\pi\(a\\mid s,z\)\\;\\;\\longrightarrow\\;\\;\\pi\(a\\mid\\ell,z\),\\qquad\\ell\\sim q\_\{\\psi\}\(\\ell\\mid s\)\.\(10\)The CIB is used consistently during data collection, training, and downstream deployment\. The CIB encoder only changes the observation channel the policy conditions on, improving robustness to shifts in global contextual information\.

## 5Experiments

![Refer to caption](https://arxiv.org/html/2607.00392v1/x3.png)\(a\)Humanoid\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x4.png)\(b\)Humanoid\-Pixels
![Refer to caption](https://arxiv.org/html/2607.00392v1/x5.png)\(c\)Quadruped\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x6.png)\(d\)Quadruped\-Pixels
![Refer to caption](https://arxiv.org/html/2607.00392v1/x7.png)\(e\)Dog\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x8.png)\(f\)Fish\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x9.png)\(g\)Kitchen\-Pixels
![Refer to caption](https://arxiv.org/html/2607.00392v1/x10.png)

Figure 3:Quantitative comparison with unsupervised skill discovery methods \(4 seeds\)\.We measure the state/task coverage of the policies\. Our algorithm scores the best coverage across all environments\. Notably, our algorithm learns meaningful skills in “Dog\-Numeric” and “Fish\-Numeric” where other methods fail\.![Refer to caption](https://arxiv.org/html/2607.00392v1/x11.png)\(a\)\(FS, RG\)
![Refer to caption](https://arxiv.org/html/2607.00392v1/x12.png)\(b\)\(RS, RG\)
![Refer to caption](https://arxiv.org/html/2607.00392v1/x13.png)\(c\)MazeEasy
![Refer to caption](https://arxiv.org/html/2607.00392v1/x14.png)\(d\)MazeHard

Figure 4:Downstream benchmark environments for Humanoid\.Prefix F\- means “Fixed” and R\- means “Random”\. Suffix \-S means “Start \(green ball in image\)”, \-G means “Goal \(red ball in image\)”\. For instance, \(FS, RG\) depicts the fixed initial start with a random goal in both the training and evaluation phases\. Maze environments serve \(RS, RG\) for the training phase, and challenging \(FS, FG\) is given for the evaluation phase\.We design our experiments to verify whether GenDa successfully overcomes the scalability bottlenecks identified in Sec\. 3\. Specifically, we investigate the following questions:

1. 1\.Data Efficiency*Does skill relabeling enhance the data efficiency of the pre\-training phase by mitigating the non\-stationarity of off\-policy learning?*
2. 2\.Generalization*Does disentangling global context via CIB produce robust skills that generalize to unseen initial conditions and downstream tasks?*
3. 3\.Scalability*Can GenDa serve as a better foundation policy for challenging, high\-dimensional control problems compared to state\-of\-the\-art baselines?*

As baselines, we includeMETRA\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3)\)andCSF\(Zhenget al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib4)\), which are state\-of\-the\-art*metric\-aware*URL methods, as well asDIAYN\(Eysenbachet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib1)\)andCIC\(Laskinet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib2)\), which represent alternative approaches to URL\. For evaluation, we use numeric and pixel\-based locomotion environments from the DeepMind Control Suite \(DMC\)\(Tassaet al\.,[2018](https://arxiv.org/html/2607.00392#bib.bib38)\), along with manipulation environments in MuJoCo\(Todorovet al\.,[2012](https://arxiv.org/html/2607.00392#bib.bib39); Schulmanet al\.,[2016](https://arxiv.org/html/2607.00392#bib.bib40); Guptaet al\.,[2020](https://arxiv.org/html/2607.00392#bib.bib41)\)\. Detailed descriptions of the environments and experimental settings for each algorithm are provided in the appendix \(see Appendix[D](https://arxiv.org/html/2607.00392#A4)\)\.

### 5\.1Main Results

##### Skill pre\-training\.

In Figure[3](https://arxiv.org/html/2607.00392#S5.F3), we evaluate the skill policy’s state and task coverage\. Coverage is measured by the number of unique coordinate bins \(or unique tasks for Kitchen\) visited acrossk=48k=48evaluation episodes using randomly sampled skills\. We set the bin size to 1 for the x and y dimensions in general environments and 0\.01 for the x, y, and z dimensions in Fish\.

In a diverse set of environments, GenDa achieves higher state coverage with substantially fewer environment interactions than prior methods\. Moreover, in*Dog*—an environment with high\-dimensional state \(226 dims\) and action \(38 dims\) spaces where existing algorithms often fail to learn meaningful skills—our method succeeds in learning skills\. These results indicate that our relabeling approach can enhance the data efficiency in pre\-training, which is also scalable to challenging tasks\.

Table 1:Quantitative comparison in downstream tasks \(4 seeds\)\.High\-level controllers are trained for 20k and 40k interaction steps in state\-based and pixel\-based environments, respectively\.
##### Downstream tasks\.

To investigate the generality and scalability of the pre\-trained policies, we propose diverse downstream tasks\.

We further evaluate on downstream tasks using the pre\-trained skill policy within a hierarchical framework\. Specifically, the skill policy is frozen, and a high\-level task policy outputs an actionzz; the skill policy then executes the corresponding skill for a fixed number of timesteps\. Each episode has a goal\. If the agent reaches the given goal, it receives a \+1 success reward and 0 reward for the other states\. The meaning of each suffix is described in Figure[4](https://arxiv.org/html/2607.00392#S5.F4)\.

\(FS, RG\) is the easiest setting, as it provides the global information distribution similar to that of the pre\-trained policy\. The \(RS, RG\) setting tests whether the skill policy can execute the learned skill under a shift in global information\. The Maze benchmark is the most challenging task in our suite; solving it requires strong generalization and exploration capabilities to perform a given skill over a wider range of coordinates\. As shown in Table[1](https://arxiv.org/html/2607.00392#S5.T1), our GenDa achieves the highest average success rate across downstream tasks\.

Table 2:Downstream performance of GenDa in Humanoid\-Numeric downstream tasks \(4 seeds\)\.We compare the performance between two pre\-trained skill policies of GenDa with different numbers of interaction steps\.Crucially, GenDa achieves comparable performance to the asymptotic limit of baselines with 5×\\timesfewer environment interactions in Table[2](https://arxiv.org/html/2607.00392#S5.T2)\. This superior data efficiency positions GenDa as a practical foundation for real\-world robotic learning where interaction is costly\.

Table 3:Performance across different environments varyingβ\\betaparameter \(3 seeds\)\.

### 5\.2GenDa ablation study

In this section, we evaluate the contribution of each component in GenDa and provide further analyses of our algorithm\.

##### Component\-wise Evaluation\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x15.png)\(a\)Skill discovery
![Refer to caption](https://arxiv.org/html/2607.00392v1/x16.png)\(b\)Downstream\(RS, RG\)
![Refer to caption](https://arxiv.org/html/2607.00392v1/x17.png)

Figure 5:Component analysis in Quadruped\-Numeric \(4 seeds\)\.We conduct ablation studies to quantify the contribution of each module in GenDa\. Figure[5](https://arxiv.org/html/2607.00392#S5.F5)demonstrates that each component addresses a distinct challenge in the URL pipeline\. Thew/o Relabvariant fails to acquire meaningful skills efficiently during the pre\-training phase\. These empirical results confirm that our skill relabeling is essential to address off\-policy non\-stationarity and maintain robustdata efficiency\. In contrast,w/o CIBachieves high state coverage during pre\-training but struggles in the \(RS, RG\) downstream task\. This discrepancy highlights that while standard objectives may increase coverage, they often lead to context\-dependent policies\. The CIB is therefore essential for disentangling the policy from the global context to ensuregeneralizability\. Collectively, these results confirm that our components are complementary: Relabeling enables efficient learning, while CIB ensures the learned skills are robust and scalable\.

##### β\\betaAnalysis\.

We conducted additional skill pre\-training experiments across diverse environments using various values ofβ\\betato verify its robustness\. The results in Table[3](https://arxiv.org/html/2607.00392#S5.T3)highlight that while performance varies depending onβ\\beta, the model is not hyper\-sensitive to the point where slight variations trigger an objective collapse in either numeric or pixel domains\. This provides evidence that our proposed objective, driven by skill relabeling and the uniformity regularizer, can be deployed across varied environments without requiring exhaustive per\-environment tuning\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x18.png)\(a\)METRA
![Refer to caption](https://arxiv.org/html/2607.00392v1/x19.png)\(b\)Ours
![Refer to caption](https://arxiv.org/html/2607.00392v1/x20.png)

Figure 6:Update\-to\-Data \(UTD\) ratio test in Humanoid\-Numeric \(4 seeds\)\.0\.125is a commonly used setting in prior work\.

### 5\.3GenDa analysis

##### Update\-to\-Data \(UTD\) Analysis\.

Figure[6](https://arxiv.org/html/2607.00392#S5.F6)serves as empirical evidence for the non\-stationary hypothesis\. While baselines suffer from instability at high UTD ratios due to semantic drift, GenDa effectively utilizes frequent updates\. This confirms that our skill relabeling turns stalezzof the replay buffer into a consistent source of supervision, enhancing the potential of off\-policy learning\. This result confirms our theoretical hypothesis in Appendix[B\.2](https://arxiv.org/html/2607.00392#A2.SS2): Variance of semantic drift in replay buffer via relabeling is the key factor enabling data\-efficient off\-policy learning\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x21.png)Figure 7:Trajectories for various offsets in Humanoid\-Numeric Environment\.Same color means the same skillzzis given to our skill policy\. An offset \(a, b\) has the same meaning as in Figure[2](https://arxiv.org/html/2607.00392#S3.F2)\.Table 4:Comparison of performance with unseen background conditions in Quadruped\-Pixels \(4 seeds\)\.’Unicolor’ and ’Gradation’ denote evaluation settings using four distinct solid colors and gradient patterns, respectively\.
##### Global information robustness\.

To assess our research question 2, we investigate how robust GenDa is to the generalization issue induced by the global context discussed in Section[3\.2](https://arxiv.org/html/2607.00392#S3.SS2)\. The qualitative results in Figure[7](https://arxiv.org/html/2607.00392#S5.F7)show GenDa has better generalizability by using CIB that encourages the skill policy to rely on the given skill and its semantic meaning rather than global context\.

##### Complementary encoder analysis\.

To investigate whether our CIB has a complementary relationship withϕ\\phi, we decode the remaining information from each representation\. Figure[8](https://arxiv.org/html/2607.00392#S5.F8)shows thatϕ\\phimainly retains background color while losing recognizable proprioception, whereas CIB retains proprioception while losing background color\.

We also tested the skill policy’s robustness by applying unseen visual shifts to the background in pixel\-based environment\. Consequently, the results in Table[4](https://arxiv.org/html/2607.00392#S5.T4)demonstrate that the CIB provides relative robustness compared to thew/o CIBin pixel environments, but its effectiveness is degraded under the visual shifts\. We abstained from using any image augmentation techniques during visual training to ensure a fair comparison with the baselines\. This highlights a highly promising direction: because the CIB module structurally disentangles redundant context, incorporating standard visual generalization techniques\(Yaratset al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib12)\)during CIB training would seamlessly bridge this gap and yield a model highly robust to visual characteristics\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x22.png)Figure 8:Remaining information of CIB encoder andϕ\\phi\.For the bottom\-right3×33\\times 3decoded images, we use the CIB decoderp​\(s^∣ℓ,ϕ​\(s\)\)p\(\\hat\{s\}\\mid\\ell,\\phi\(s\)\)\. For the decoded images ofℓi\\ell\_\{i\}andϕi\\phi\_\{i\}, we train independent decoders that do not affect the base encoder\.

## 6Related Work

Unsupervised Skill DiscoveryUnsupervised reinforcement learning \(URL\) aims to learn skill representations, policies, or dynamics models through interaction with the environment without any external task reward\(Eysenbachet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib1); Warde\-Farleyet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib15); Camposet al\.,[2020](https://arxiv.org/html/2607.00392#bib.bib16); Hartikainenet al\.,[2020](https://arxiv.org/html/2607.00392#bib.bib17)\), and then reuse them for downstream learning\(Zhaoet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib19); Laskinet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib2); Yanget al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib29)\)\. Prior unsupervised skill discovery methods train a skill\-conditioned policy from states or trajectories by maximizing mutual information objectives\(Eysenbachet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib1); Laskinet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib2); Strouseet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib20)\)\. They enable the learning of diverse and distinguishable behaviors even without external rewards\. Some approaches further leverage dynamics models to capture mutual information at the state\-transition level\(Sharmaet al\.,[2020](https://arxiv.org/html/2607.00392#bib.bib13); Mendoncaet al\.,[2021](https://arxiv.org/html/2607.00392#bib.bib21)\)\. However, these methods can prioritize discriminability over how far behaviors spread in the environment\. As a result, in high\-dimensional state spaces, they have been reported to discover redundant or locally confined skills\(Parket al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib22),[2024](https://arxiv.org/html/2607.00392#bib.bib3); Zhenget al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib4)\)\. To mitigate this, more recent advances in URL leverage explicit state\-space metrics to discover skills with Wasserstein dependency measure\(Ozairet al\.,[2019](https://arxiv.org/html/2607.00392#bib.bib24); Heet al\.,[2022](https://arxiv.org/html/2607.00392#bib.bib23)\)and InfoNCE\(van den Oordet al\.,[2018](https://arxiv.org/html/2607.00392#bib.bib25); Ma and Collins,[2018](https://arxiv.org/html/2607.00392#bib.bib26); Henaff,[2020](https://arxiv.org/html/2607.00392#bib.bib27); Myerset al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib28)\)that transfer better to downstream tasks\(Parket al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib3); Zhenget al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib4)\)\. Despite the promise of URL, prior methods face two limitations: \(1\) data efficiency and \(2\) policy generalization\.

Disentanglement in unsupervised skill discovery\.Some prior work has explored inducing disentanglement by explicitly structuring the latent space via factorization\(Huet al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib30); Wanget al\.,[2024](https://arxiv.org/html/2607.00392#bib.bib31)\)\. Another line of work proposes masking the policy input to encourage structured skill usage\(Parket al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib32)\)\. However, these methods tend to rely on assumptions such as factorized factors or periodic structure\. In contrast, our complementary variational encoder \(CIB\) improves skill reusability by modulating the skill policy’s input conditioning, without a hand\-crafted structure\.

## 7Conclusion

### 7\.1Implication for Unsupervised Skill Learning

We identify two fundamental challenges in off\-policy unsupervised reinforcement learning that limit the scalability of skill discovery\. Our skill relabeling mitigates the semantic drift in prior URL methods together with a compatible regularization term\. It significantly improves the data efficiency in the pre\-training phase\. We also point out a failure mode of the skill policy in downstream tasks, which is caused by global contextual information\. With our drop\-in CIB module, the generalizability of the skill policy is improved in various downstream tasks\.

### 7\.2Open Challenges and Future Directions

##### Skill relabeling

Our skill relabeling still inherits a*one\-skill\-per\-episode*assumption that can be violated when a single episode contains heterogeneous behaviors \(e\.g\., exploratory “moving around” segments interleaved with skill\-consistent transitions\)\. Our relabeling target is derived from the telescoping sum; thus, it can be generalized to any arbitrary time segment\. So, our skill relabeling framework is structurally and mathematically equipped to handle intra\-episode skill changes\. We will explore extending relabeling to assigning time\-varying skill labels, which may better capture abrupt changes in intent, such as sharp direction switches or rapid reactive adjustments\.

##### Generalization

CIB is designed to disentangle the global context from the skill policy; its effectiveness depends on the learned metric\-aware latent functionϕ\\phi; if one wishes to apply the idea to non\-metric\-aware skill discovery algorithms, this dependency can limit applicability\. We consider it a feasible future direction to use*dynamics\-aware*objectives rather than a particular embedding metric\.

##### Manipulation task

Our metric\-aware objective, which maximizes temporal state distances, inherently encourages large\-scale movements\. This objective hinders the acquisition of the fine\-grained, small\-scale behaviors required for precise manipulation\. Addressing this will require incorporating magnitude\-conditioned skill mechanisms, such as PSD\(Parket al\.,[2025](https://arxiv.org/html/2607.00392#bib.bib32)\), to balance large explorations with delicate object manipulation\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## Acknowledgements

This work was partly supported by the National Research Foundation of Korea \(NRF\) \(No\. RS\-2024\-00348376\), by the Institute of Information&\\&Communications Technology Planning&\\&Evaluation \(IITP\) \(No\. RS\-2024\-00438686, No\. RS\-2022\-II221045, No\. RS\-2025\-02218768, No\. RS\-2019\-II190421\) grant funded by the Korea government \(MSIT\), and by the Industrial Technology Innovation Program \(No\. RS\-2025\-25448266\) funded by the Ministry of Trade, Industry&\\&Energy \(MOTIE\) and Korea Evaluation Institute of Industrial Technology \(KEIT\)\.

## References

- A\. A\. Alemi, I\. Fischer, J\. V\. Dillon, and K\. Murphy \(2017\)Deep variational information bottleneck\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.3\.1](https://arxiv.org/html/2607.00392#S4.SS3.SSS1.p2.3)\.
- M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. Zaremba \(2017\)Hindsight experience replay\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[2nd item](https://arxiv.org/html/2607.00392#S1.I1.i2.p1.1),[§4\.2](https://arxiv.org/html/2607.00392#S4.SS2.p3.6)\.
- V\. Campos, A\. Trott, C\. Xiong, R\. Socher, X\. Giro\-i\-Nieto, and J\. Torres \(2020\)Explore, discover and learn: unsupervised discovery of state\-covering skills\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- B\. Eysenbach, A\. Gupta, J\. Ibarz, and S\. Levine \(2019\)Diversity is all you need: learning skills without a reward function\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p3.5),[§1](https://arxiv.org/html/2607.00392#S1.p4.1),[§5](https://arxiv.org/html/2607.00392#S5.p2.1),[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- K\. Gregor, D\. Jimenez Rezende, and D\. Wierstra \(2016\)Variational intrinsic control\.arXiv preprint arXiv:1611\.07507\.Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- A\. Gupta, V\. Kumar, C\. Lynch, S\. Levine, and K\. Hausman \(2020\)Relay policy learning: solving long\-horizon tasks via imitation and reinforcement learning\.InConference on Robot Learning \(CoRL\),Cited by:[§5](https://arxiv.org/html/2607.00392#S5.p2.1)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p3.5),[13](https://arxiv.org/html/2607.00392#algx1.l13)\.
- K\. Hartikainen, X\. Geng, T\. Haarnoja, and S\. Levine \(2020\)Dynamical distance learning for semi\-supervised and unsupervised skill discovery\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- S\. He, Y\. Jiang, H\. Zhang, J\. Shao, and X\. Ji \(2022\)Wasserstein unsupervised reinforcement learning\.InAssociation for the Advancement of Artificial Intelligence \(AAAI\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- O\. Henaff \(2020\)Data\-efficient image recognition with contrastive predictive coding\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- J\. Hu, Z\. Wang, P\. Stone, and R\. Martín\-Martín \(2024\)Disentangled unsupervised skill discovery for efficient hierarchical reinforcement learning\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p2.1)\.
- P\. Kamienny, J\. Tarbouriech, S\. Lamprier, A\. Lazaric, and L\. Denoyer \(2022\)Direct then diffuse: incremental unsupervised skill discovery for state covering and goal reaching\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- J\. Kim, S\. Park, and G\. Kim \(2021\)Unsupervised skill discovery with bottleneck option learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- M\. Laskin, H\. Liu, X\. B\. Peng, D\. Yarats, A\. Rajeswaran, and P\. Abbeel \(2022\)Unsupervised reinforcement learning with contrastive intrinsic control\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p3.5),[§5](https://arxiv.org/html/2607.00392#S5.p2.1),[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- M\. Laskin, D\. Yarats, H\. Liu, K\. Lee, A\. Zhan, K\. Lu, C\. Cang, L\. Pinto, and P\. Abbeel \(2021\)URLB: unsupervised reinforcement learning benchmark\.InNeural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- Z\. Ma and M\. Collins \(2018\)Noise contrastive estimation and negative sampling for conditional models: consistency and statistical efficiency\.InEmpirical Methods in Natural Language Processing \(EMNLP\),pp\. 3698–3707\.Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- R\. Mendonca, O\. Rybkin, K\. Daniilidis, D\. Hafner, and D\. Pathak \(2021\)Discovering and achieving goals via world models\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- V\. Myers, C\. Zheng, A\. Dragan, S\. Levine, and B\. Eysenbach \(2024\)Learning temporal distances: contrastive successor features can provide a metric structure for decision\-making\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- S\. Ozair, C\. Lynch, Y\. Bengio, A\. van den Oord, S\. Levine, and P\. Sermanet \(2019\)Wasserstein dependency measure for representation learning\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- J\. Park, D\. Cho, J\. Lee, D\. Shim, I\. Jang, and H\. J\. Kim \(2025\)Periodic skill discovery\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p2.1),[§7\.2](https://arxiv.org/html/2607.00392#S7.SS2.SSS0.Px3.p1.1)\.
- S\. Park, J\. Choi, J\. Kim, H\. Lee, and G\. Kim \(2022\)Lipschitz\-constrained unsupervised skill discovery\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- S\. Park, K\. Lee, Y\. Lee, and P\. Abbeel \(2023\)Controllability\-aware unsupervised skill discovery\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- S\. Park, O\. Rybkin, and S\. Levine \(2024\)METRA: scalable unsupervised rl with metric\-aware abstraction\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p3.5),[§1](https://arxiv.org/html/2607.00392#S1.p4.1),[Figure 1](https://arxiv.org/html/2607.00392#S2.F1.2.1),[Figure 1](https://arxiv.org/html/2607.00392#S2.F1.4.2),[§2](https://arxiv.org/html/2607.00392#S2.SS0.SSS0.Px1.p1.7),[§2](https://arxiv.org/html/2607.00392#S2.SS0.SSS0.Px1.p3.2),[Figure 2](https://arxiv.org/html/2607.00392#S3.F2),[Figure 2](https://arxiv.org/html/2607.00392#S3.F2.4.2.1),[§4\.3\.1](https://arxiv.org/html/2607.00392#S4.SS3.SSS1.p1.3),[§4](https://arxiv.org/html/2607.00392#S4.p1.1),[§5](https://arxiv.org/html/2607.00392#S5.p2.1),[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- S\. Rajeswar, P\. Mazzaglia, T\. Verbelen, A\. Piché, B\. Dhoedt, A\. C\. Courville, and A\. Lacoste \(2023\)Mastering the unsupervised reinforcement learning benchmark from pixels\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- J\. Schulman, P\. Moritz, S\. Levine, M\. I\. Jordan, and P\. Abbeel \(2016\)High\-dimensional continuous control using generalized advantage estimation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5](https://arxiv.org/html/2607.00392#S5.p2.1)\.
- M\. Schwarzer, A\. Anand, R\. Goel, R\. D\. Hjelm, A\. Courville, and P\. Bachman \(2021\)Data\-efficient reinforcement learning with self\-predictive representations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.1\.1](https://arxiv.org/html/2607.00392#S4.SS1.SSS1.Px3.p1.1)\.
- A\. Sharma, S\. Gu, S\. Levine, V\. Kumar, and K\. Hausman \(2020\)Dynamics\-aware unsupervised discovery of skills\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p3.5),[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- D\. Strouse, K\. Baumli, D\. Warde\-Farley, V\. Mnih, and S\. S\. Hansen \(2022\)Learning more skills through optimistic exploration\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq, T\. Lillicrap, and M\. Riedmiller \(2018\)DeepMind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§5](https://arxiv.org/html/2607.00392#S5.p2.1)\.
- E\. Todorov, T\. Erez, and Y\. Tassa \(2012\)MuJoCo: a physics engine for model\-based control\.InIEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),Cited by:[§5](https://arxiv.org/html/2607.00392#S5.p2.1)\.
- A\. van den Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§4\.1\.2](https://arxiv.org/html/2607.00392#S4.SS1.SSS2.p3.5),[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- Z\. Wang, J\. Hu, C\. Chuck, S\. Chen, R\. Martín\-Martín, A\. Zhang, S\. Niekum, and P\. Stone \(2024\)SkiLD: unsupervised skill discovery guided by factor interactions\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p2.1)\.
- D\. Warde\-Farley, T\. Van de Wiele, T\. Kulkarni, C\. Ionescu, S\. Hansen, and V\. Mnih \(2019\)Unsupervised control through non\-parametric discriminative rewards\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- R\. Yang, C\. Bai, H\. Guo, S\. Li, B\. Zhao, Z\. Wang, P\. Liu, and X\. Li \(2023\)Behavior contrastive learning for unsupervised skill discovery\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p1.1)\.
- Y\. Yang, T\. Zhou, Q\. He, L\. Han, M\. Pechenizkiy, and M\. Fang \(2024\)Task adaptation from skills: information geometry, disentanglement, and new objectives for unsupervised reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- D\. Yarats, R\. Fergus, A\. Lazaric, and L\. Pinto \(2022\)Mastering visual continuous control: improved data\-augmented reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5\.3](https://arxiv.org/html/2607.00392#S5.SS3.SSS0.Px3.p2.1)\.
- A\. Zhao, M\. G\. Lin, Y\. Li, Y\. Liu, and G\. Huang \(2022\)A mixture of surprises for unsupervised reinforcement learning\.InNeural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.
- C\. Zheng, J\. Tuyls, J\. Peng, and B\. Eysenbach \(2025\)Can a misl fly? analysis and ingredients for mutual information skill learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.00392#S1.p3.5),[§1](https://arxiv.org/html/2607.00392#S1.p4.1),[§5](https://arxiv.org/html/2607.00392#S5.p2.1),[§6](https://arxiv.org/html/2607.00392#S6.p1.1)\.

## Appendix ARelabel\-and\-Optimize Procedure

##### Setting\.

LetBBbe a fixed replay buffer containing either transitions\(s,s′,z\)\(s,s^\{\\prime\},z\)or trajectoriesτ\\tauwith endpoints\(s0​\(τ\),sT​\(τ\)\)\(s\_\{0\}\(\\tau\),s\_\{T\}\(\\tau\)\)\. LetV⊆𝒮V\\subseteq\\mathcal\{S\}denote the finite set of states that appear in the dataset \(tabular setting\), and restrict the optimization variable to

ϕ:V→ℝd\.\\phi:V\\to\\mathbb\{R\}^\{d\}\.Henceϕ\\phican be identified with a vector in the finite\-dimensional spaceℝd​\|V\|\\mathbb\{R\}^\{d\|V\|\}\.

We encode the Lipschitz constraints via an undirected graphG=\(V,E\)G=\(V,E\):

\{u,v\}∈E⇔a transition​u→v​or​v→u​is observed in the dataset\.\\\{u,v\\\}\\in E\\iff\\text\{a transition \}u\\to v\\text\{ or \}v\\to u\\text\{ is observed in the dataset\}\.We fix the gauge by choosing an anchor states0∈Vs\_\{0\}\\in Vand enforcingϕ​\(s0\)=0\\phi\(s\_\{0\}\)=0\.

###### Assumption A\.1\(Support\-connectedness / gauge fixing\)\.

All states that appear in theϕ\\phi\-step objective belong to the connected componentC​\(s0\)C\(s\_\{0\}\)of the anchors0s\_\{0\}inGG\. Equivalently, restrictingGGtoC​\(s0\)C\(s\_\{0\}\)yields a connected graph\. \(Alternatively, one may place one anchor per connected component; we use this assumption for notational simplicity\.\)

##### Relabeling map\.

Forε\>0\\varepsilon\>0, define

g​\(ϕ;a,b\)=ϕ​\(b\)−ϕ​\(a\)max⁡\(‖ϕ​\(b\)−ϕ​\(a\)‖,ε\)\.g\(\\phi;a,b\)\\;=\\;\\frac\{\\phi\(b\)\-\\phi\(a\)\}\{\\max\(\\\|\\phi\(b\)\-\\phi\(a\)\\\|,\\ \\varepsilon\)\}\.Thenggis well\-defined for all\(a,b\)\(a,b\)and continuous, with‖g​\(ϕ;a,b\)‖≤1\\\|g\(\\phi;a,b\)\\\|\\leq 1\. Whileggmay not be differentiable on the boundary‖ϕ​\(b\)−ϕ​\(a\)‖=ε\\\|\\phi\(b\)\-\\phi\(a\)\\\|=\\varepsilon, differentiability is not required for the existence results below\.

##### Relabel\-and\-optimize procedure\.

Given the currentϕ\\phi,

1. 1\.\(z\-step; relabel\)For each trajectoryτ\\tau, set zτ←g​\(ϕ;s0​\(τ\),sT​\(τ\)\)\.z\_\{\\tau\}\\leftarrow g\(\\phi;\\ s\_\{0\}\(\\tau\),\\ s\_\{T\}\(\\tau\)\)\.
2. 2\.\(ϕ\\phi\-step; optimize\)Treat\{zτ\}\\\{z\_\{\\tau\}\\\}as constants during theϕ\\phi\-update \(i\.e\., apply a stop\-gradient throughzτz\_\{\\tau\}\) and solve maxϕ\\displaystyle\\max\_\{\\phi\}\\quad∑τ⟨ϕ​\(sT​\(τ\)\)−ϕ​\(s0​\(τ\)\),zτ⟩\\displaystyle\\sum\_\{\\tau\}\\left\\langle\\phi\\big\(s\_\{T\}\(\\tau\)\\big\)\-\\phi\\big\(s\_\{0\}\(\\tau\)\\big\),\\ z\_\{\\tau\}\\right\\rangle\(11\)s\.t\.‖ϕ​\(u\)−ϕ​\(v\)‖≤1∀\{u,v\}∈E,\\displaystyle\\\|\\phi\(u\)\-\\phi\(v\)\\\|\\leq 1\\quad\\forall\\\{u,v\\\}\\in E,\(12\)ϕ​\(s0\)=0\.\\displaystyle\\phi\(s\_\{0\}\)=0\.\(13\)

We emphasize that we do not claim global monotonic improvement or convergence of the full iteration without additional structure; instead, we establish that each step is well\-defined and that theϕ\\phi\-step admits an optimizer at every iteration\.

###### Proof Sketch\.

Feasibility\.The feasible set is non\-empty sinceϕ​\(s\)≡0\\phi\(s\)\\equiv 0for alls∈Vs\\in Vsatisfies \([12](https://arxiv.org/html/2607.00392#A1.E12)\) and \([13](https://arxiv.org/html/2607.00392#A1.E13)\)\.

Boundedness\.Fix anys∈C​\(s0\)s\\in C\(s\_\{0\}\)\. Lets0=v0,v1,…,vL=ss\_\{0\}=v\_\{0\},v\_\{1\},\\dots,v\_\{L\}=sbe a shortest path inGG\. By the triangle inequality and the Lipschitz constraints,

‖ϕ​\(s\)‖=‖ϕ​\(vL\)−ϕ​\(v0\)‖≤∑i=1L‖ϕ​\(vi\)−ϕ​\(vi−1\)‖≤∑i=1L1=L=distG​\(s0,s\)\.\\\|\\phi\(s\)\\\|=\\\|\\phi\(v\_\{L\}\)\-\\phi\(v\_\{0\}\)\\\|\\leq\\sum\_\{i=1\}^\{L\}\\\|\\phi\(v\_\{i\}\)\-\\phi\(v\_\{i\-1\}\)\\\|\\leq\\sum\_\{i=1\}^\{L\}1=L=\\mathrm\{dist\}\_\{G\}\(s\_\{0\},s\)\.Thus each coordinate blockϕ​\(s\)\\phi\(s\)is bounded over the feasible set\.

Compactness\.SinceVVis finite,ϕ\\philies in a finite\-dimensional Euclidean space\. Constraints \([12](https://arxiv.org/html/2607.00392#A1.E12)\)–\([13](https://arxiv.org/html/2607.00392#A1.E13)\) define a closed set \(inverse images of continuous functions\), and the feasible set is bounded by the previous step; hence it is compact \(Heine–Borel\)\.

Existence\.The objective \([11](https://arxiv.org/html/2607.00392#A1.E11)\) is linear \(hence continuous\) inϕ\\phiand the feasible set is compact, so an optimizer exists by the Weierstrass theorem\.

Global optimality\.The objective is linear \(thus concave\) and the constraints are convex, so this is a convex optimization problem\. Therefore any optimizer is globally optimal\. ∎

###### Proposition A\.2\(Existence and global optimality of theϕ\\phi\-step solution\)\.

Under Assumption[A\.1](https://arxiv.org/html/2607.00392#A1.Thmtheorem1)\(tabular setting\), for any fixed relabeled signals\{zτ\}\\\{z\_\{\\tau\}\\\}, theϕ\\phi\-step problem \([11](https://arxiv.org/html/2607.00392#A1.E11)\)–\([13](https://arxiv.org/html/2607.00392#A1.E13)\) admits at least one optimal solutionϕ⋆\\phi^\{\\star\}\. Moreover, the problem is a convex optimization problem, hence any optimizer is globally optimal\.

###### Corollary A\.3\(Well\-posedness of each iteration\)\.

Forε\>0\\varepsilon\>0, the relabeling map is well\-defined, and by Proposition[A\.2](https://arxiv.org/html/2607.00392#A1.Thmtheorem2)theϕ\\phi\-step admits an optimizer at every iteration\. Hence the relabel\-and\-optimize procedure is well\-posed\.

## Appendix BRelabeling under Non\-Stationary Label Noise

### B\.1Non\-Stationary with a fixed z label\.

##### Noise model

For each trajectoryτ\\tauwith endpoints\(s0​\(τ\),sT​\(τ\)\)\(s\_\{0\}\(\\tau\),s\_\{T\}\(\\tau\)\), suppose the original labelzτz\_\{\\tau\}is noisy:

𝔼​\[zτ∣s0,sT\]=μ​\(s0,sT\),Var​\(zτ∣s0,sT\)=Σ​\(s0,sT\)\.\\mathbb\{E\}\[z\_\{\\tau\}\\mid s\_\{0\},s\_\{T\}\]=\\mu\(s\_\{0\},s\_\{T\}\),\\qquad\\mathrm\{Var\}\(z\_\{\\tau\}\\mid s\_\{0\},s\_\{T\}\)=\\Sigma\(s\_\{0\},s\_\{T\}\)\.In non\-stationary settings, the conditional varianceΣ​\(s0,sT\)\\Sigma\(s\_\{0\},s\_\{T\}\)may vary across endpoint pairs, and typicallyΣ​\(s0,sT\)≠0\\Sigma\(s\_\{0\},s\_\{T\}\)\\neq 0on parts of the support\.

##### Alignment scalar and conditional variability\.

For a fixedϕ\\phi, define the per\-trajectory alignment contribution

Yτ​\(ϕ\)=⟨ϕ​\(sT​\(τ\)\)−ϕ​\(s0​\(τ\)\),zτ⟩\.Y\_\{\\tau\}\(\\phi\)\\;=\\;\\left\\langle\\phi\\big\(s\_\{T\}\(\\tau\)\\big\)\-\\phi\\big\(s\_\{0\}\(\\tau\)\\big\),\\ z\_\{\\tau\}\\right\\rangle\.WhenΣ​\(s0,sT\)≠0\\Sigma\(s\_\{0\},s\_\{T\}\)\\neq 0, the conditional variance ofYτ​\(ϕ\)Y\_\{\\tau\}\(\\phi\)generally remains positive:

Var​\(Yτ​\(ϕ\)∣s0,sT,ϕ\)\>0,\\mathrm\{Var\}\\big\(Y\_\{\\tau\}\(\\phi\)\\mid s\_\{0\},s\_\{T\},\\phi\\big\)\>0,reflecting that even for the same endpoint pair, label noise induces variability in the alignment term\.

### B\.2zzrelabeling and label noise\.

##### Relabeling removes conditional label\-noise variability \(from theϕ\\phi\-step perspective\)\.

Define the relabeled signal deterministically via

z~τ=g​\(ϕ;s0​\(τ\),sT​\(τ\)\),\\tilde\{z\}\_\{\\tau\}\\;=\\;g\(\\phi;\\ s\_\{0\}\(\\tau\),s\_\{T\}\(\\tau\)\),and in theϕ\\phi\-step treatz~τ\\tilde\{z\}\_\{\\tau\}as a constant \(stop\-gradient\)\. Then the relabeled alignment term

Y~τ​\(ϕ\)=⟨ϕ​\(sT​\(τ\)\)−ϕ​\(s0​\(τ\)\),z~τ⟩\\tilde\{Y\}\_\{\\tau\}\(\\phi\)\\;=\\;\\left\\langle\\phi\\big\(s\_\{T\}\(\\tau\)\\big\)\-\\phi\\big\(s\_\{0\}\(\\tau\)\\big\),\\ \\tilde\{z\}\_\{\\tau\}\\right\\ranglesatisfies

Var​\(Y~τ​\(ϕ\)∣s0,sT,ϕ\)=0\.\\mathrm\{Var\}\\big\(\\tilde\{Y\}\_\{\\tau\}\(\\phi\)\\mid s\_\{0\},s\_\{T\},\\phi\\big\)=0\.Thus, from the perspective of theϕ\\phi\-step update, conditional variability due to label noise is removed\.

##### Implication: potential reduction in stochastic\-gradient variance \(with possible bias\)\.

Because the per\-sample gradient of \([11](https://arxiv.org/html/2607.00392#A1.E11)\) is affine \(here linear\) in the labelzz, the law of total variance suggests decomposing stochastic\-gradient variance into \(i\) a component attributable to conditional label noise and \(ii\) a component attributable to sampling\. When using the original noisy labels, the former component persists wheneverΣ​\(s0,sT\)≠0\\Sigma\(s\_\{0\},s\_\{T\}\)\\neq 0\. When using relabeling \(and treatingz~τ\\tilde\{z\}\_\{\\tau\}as constant in theϕ\\phi\-step\), this conditional label\-noise component can drop to zero, potentially reducing gradient variance and improving update stability in non\-stationary regimes\.

However, sincez~τ\\tilde\{z\}\_\{\\tau\}depends onϕ\\phi, relabeling can introduce bias relative to updates driven by the original noisy labels\. Hence relabeling reflects a bias–variance trade\-off: it may improve stability when label noise dominates, provided the induced bias remains controlled\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x23.png)\(a\)Ours w/oℒu​n​i​f\\mathcal\{L\}\_\{unif\}
![Refer to caption](https://arxiv.org/html/2607.00392v1/x24.png)\(b\)Ours wℒu​n​i​f\\mathcal\{L\}\_\{unif\}
Left of each pair: \(τz\\tau\_\{z\}\), Right of each pair: \(ϕ​\(τz\)\\phi\(\\tau\_\{z\}\)\)

Figure 9:Trajectoryτz\\tau\_\{z\}and its latentϕ​\(τz\)\\phi\(\\tau\_\{z\}\)in Humanoid\-Numeric at 1M timesteps\.τz\\tau\_\{z\}is azz\-conditioned trajectory, and the same color indicates the same skillzzis given\. Our uniformity term can encourage the latent space to have diverse directions spread in the early step of learning\.

## Appendix CPolicy Relabeling Analysis\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x25.png)Figure 10:Skill policy intrinsic reward z in Quadruped\-Numeric\(4 seeds\)\.zr​o​l​lz\_\{roll\}: Using fixed z only,c: c\-step relabel,τ\\tau:0→T0\\rightarrow Trelabel,Random z: newly sampled random z,Ours: Mix\(c,τ\\tau,zr​o​l​lz\_\{roll\}\)To investigate the properzzconditions for the intrinsic reward function, we evaluate multiple relabeling strategies that can be used for selecting targetzzand report their comparative results in Figure[10](https://arxiv.org/html/2607.00392#A3.F10)\.

For Mix\(c,τ\\tau\), we use a 0\.5:0\.5 ratio\. For Ours, we compute the probability ofzrollz\_\{\\text\{roll\}\}using a count\-based estimator with respect to thezrelabz\_\{\\text\{relab\}\}directions of the episodes stored in the replay buffer\. This estimator is updated like an exponential moving average \(EMA\) each time azrollz\_\{\\text\{roll\}\}is sampled from the buffer for learning\. If the resulting probability is lower than thresholdϵz=0\.4\\epsilon\_\{z\}=0\.4, we keep the correspondingzrollz\_\{\\text\{roll\}\}; otherwise, we relabel it usingMix​\(c,τ\)\\mathrm\{Mix\}\(c,\\tau\)\. We also test multiple settings for the c\-τ\\taumixture ratio and the thresholdϵz\\epsilon\_\{z\}in Table[5](https://arxiv.org/html/2607.00392#A3.T5), but observe only marginal differences in performance\.

Table 5:State coverage corresponding to different mixture ratios and thresholds in Quadruped\-Numeric\(3 seeds\)\.

## Appendix DImplementation details

We implement our algorithm and baselines with JAX for faster training\. Our code is based on the official PyTorch implementation and paper of each baseline\. For stable learning, we use the LayerNorm in the critic networks of both our method and the baselines\. For CIC and DIAYN, we use the hyperparameters used in the official CSF implementation\. For METRA, CSF and Ours, we use same hyperparameters except high\-level controllers in downstream tasks that are represented in Table[8](https://arxiv.org/html/2607.00392#A4.T8)\.

![Refer to caption](https://arxiv.org/html/2607.00392v1/x26.png)\(a\)Quadruped\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x27.png)\(b\)Dog\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x28.png)\(c\)Fish\-Numeric
![Refer to caption](https://arxiv.org/html/2607.00392v1/x29.png)\(d\)Humanoid and Goal
![Refer to caption](https://arxiv.org/html/2607.00392v1/x30.png)\(e\)Humanoid\-Pixels
![Refer to caption](https://arxiv.org/html/2607.00392v1/x31.png)\(f\)Quadruped\-Pixels
![Refer to caption](https://arxiv.org/html/2607.00392v1/x32.png)\(g\)Kitchen\-Pixels

Figure 11:Benchmark environments for various domains\.### D\.1Environment settings

Table 6:Environment parameters\.##### Downstream tasks\.

To evaluate performance on downstream tasks, we utilize 6 domains, each containing 1 to 4 specific tasks\. High\-level controllersπh​\(z\|s,st​a​s​k\)\\pi^\{h\}\(z\|s,s^\{task\}\)selects a skill every K=25 environment steps, wherest​a​s​ks^\{task\}denotes the task information \(like desired goal\)\.

Maze tasks\.The agent navigates a7×77\\times 7maze to reach a goal\. During training, both the agent and the goal are initialized at random feasible positions, with a minimum separation distance of 3\. During evaluation, the agent starts from a fixed location\(1,0\)\(1,0\)\. The goal is placed at\(5,6\)\(5,6\)forEasyand at\(1,6\)\(1,6\)forHard\. For the Quadruped agent, we scale the maze dimensions by a factor of1\.51\.5to account for its larger body size\. We set the episode horizon to 400 environment steps forEasyand 500 steps forHard\.

\(RS, RG\) tasks \(Quadruped, Humanoid, Dog\-Numeric\)\.Both the agent and the goal are initialized uniformly at random in\[−3\.5,3\.5\]2\[\-3\.5,3\.5\]^\{2\}, with a minimum separation distance of 3\. As in the Maze tasks, we increase the environment scale by a factor of1\.51\.5for the Quadruped agent\.

\(FS, RG\) tasks\.The agent starts from the origin\(0,0\)\(0,0\), while the goal is sampled randomly\. Specifically, the goal is sampled from\[−3\.5,3\.5\]2\[\-3\.5,3\.5\]^\{2\}for Humanoid\-Numeric and from\[−5,5\]2\[\-5,5\]^\{2\}for Humanoid and Quadruped\-Pixels\. In the Fish environment, the agent starts from a fixed position with a random orientation \(quaternion\), and the goal is placed 0\.5 units ahead along the agent’s forward direction\. As in the Maze tasks, we increase the environment scale by a factor of1\.51\.5for the Quadruped agent\.

For \(FS, RG\) and \(RS, RG\), we set the episode horizon to 200 environment steps forQuadruped \(Num/Pix\),Humanoid \(Num/Pix\), andFish, and to 400 steps forDog\-Numeric\.

Table 7:Hyperparameters for unsupervised skill discovery methods\.Table 8:Hyperparameters for SAC in high\-level controllers\.