Active Curriculum Refinement for Reinforcement Learning

# Active Curriculum Refinement for Reinforcement Learning
Source: [https://arxiv.org/html/2608.26469](https://arxiv.org/html/2608.26469)
Zhenya LiuYuxin ChenAffiliation:The University of Chicago, Chicago IL, USACorrespondence to:[chenyuxin@uchicago\.edu](mailto:[email protected])

###### Abstract

In many RL domains, environments are linked by prerequisite relations—e\.g\., difficulty\-increasing edits or parameter increments—which induce a directed acyclic curriculum graph \(DAG\)\. In practice, this structure is often exploited only implicitly, yet it can yield clear gains in training\. We introducePATH, a curriculum learning framework that performs active learning on the curriculum graph\.PATHfirst expands coverage by sampling diverse curriculum paths, then reallocates training toward regions that remain unmastered\. Experiments show thatPATHleverages the graph structure to achieve strong robustness and generalization across diverse environments\.

###### Keywords:

Reinforcement Learning, Curriculum Learning, Unsupervised Environment Design, PLR, ACCEL, Scheduling

## 1Introduction

Deep reinforcement learning \(RL\) is a central paradigm in modern AI, underpinning successes in complex decision\-making problems\([Silver et al\., 2016](https://arxiv.org/html/2608.26469#bib.bib27)\)and post\-training for instruction\-following LLMs\([Ouyang et al\., 2022](https://arxiv.org/html/2608.26469#bib.bib26)\)\. As RL systems move beyond single benchmark instances, a core challenge is generalization: policies trained on a finite set of environments often overfit and fail to transfer robustly across a broader environment distribution\([Cobbe et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib28)\)\. Curriculum learning\([Bengio et al\., 2009](https://arxiv.org/html/2608.26469#bib.bib1);[Wang et al\., 2022](https://arxiv.org/html/2608.26469#bib.bib3)\)offers a practical approach for RL generalization by controlling which environments are visited and in what order\([Narvekar et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib2);[Parker\-Holder et al\., 2022b](https://arxiv.org/html/2608.26469#bib.bib29);[Soviany et al\., 2022](https://arxiv.org/html/2608.26469#bib.bib30)\), and recent work increasingly focuses on autonomously generated curricula, including unsupervised environment design \(UED\)\([Dennis et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib9)\)\. While fully automatic curriculum generation is often emphasized, many effective methods do not build curricula from scratch; instead, they implicitly exploit pre\-existing structure in the environment distribution—e\.g\., an inherent easier\-to\-harder ordering that supports curriculum progression\([Florensa et al\., 2017](https://arxiv.org/html/2608.26469#bib.bib5);[Parker\-Holder et al\., 2022a](https://arxiv.org/html/2608.26469#bib.bib11)\)\.

Crucially, in many RL settings such curriculum orderings are straightforward to specify, and can be made explicit in three practical ways\. \(i\)Parameterized increments:environments expose interpretable parameters, so successor environments can be generated by small increases along a chosen parameter\([Wang et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib22);[Tidd et al\., 2021](https://arxiv.org/html/2608.26469#bib.bib32)\)\. \(ii\)Complexity\-increasing edits:successor environments are obtained by local edits that make the instance more complex, e\.g\., adding obstacles\([Li et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib50);[Parker\-Holder et al\., 2022a](https://arxiv.org/html/2608.26469#bib.bib11)\)\. \(iii\)Feature\-induced structure:when parameters or edits are unavailable, instances can be represented by descriptors or embeddings, and edges follow monotone progress along one or multiple feature axes\([Lee et al\., 2025](https://arxiv.org/html/2608.26469#bib.bib12)\)\. In all three cases, the environment space provides usable curriculum orderings, naturally inducing a directed acyclic graph \(DAG\) over instances; training along a single ordering often yields an effective curriculum progression\.

This raises a natural question: given an explicit curriculum DAG, can we allocate training efficiently so that traversing relatively few paths suffices to master the entire reachable environment space? We formalize this as a new curriculum learning framework—*active learning on the curriculum graph*\. We introducePATH, a two\-stage automated and adaptive algorithm:PATH:Randomsamples curriculum paths randomly to drive broad exploration and expand coverage, andPATH:Activeuses regret\-based signals to identify high learning\-potential regions and reallocates budget by sampling more paths through their successor subgraphs\. Overall,PATHtreats curriculum learning as active path acquisition on a curriculum DAG, enabling efficient mastery of the reachable environment space\. In summary, our contributions are:

- •We formalize curriculum learning over structured environment families as active learning on a curriculum DAG, where training corresponds to acquiring informative paths rather than isolated instances\.
- •We introducePATH, a lightweight two\-stage algorithm that combines random path exploration for coverage with regret\-driven successor expansion to efficiently allocate training budget\.
- •We demonstrate on both discrete \(MiniGrid\) and continuous\-control \(BipedalWalker\) benchmarks that exploiting curriculum graph structure yields substantially improved robustness and generalization over prior curriculum and regret\-based methods\.

## 2Preliminaries

### 2\.1Reinforcement Learning

##### Environment\.

An environment is a discounted Markov decision process \(MDP\)ℳ=\(𝒮,𝒜,P,r,ρ0,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\rho\_\{0\},\\gamma\), where𝒮\\mathcal\{S\}is the state space,𝒜\\mathcal\{A\}is the action space,P\(⋅∣s,a\)P\(\\cdot\\mid s,a\)is the transition kernel,r⁡\(s,a,s′\)r\(s,a,s^\{\\prime\}\)is the reward function,ρ0\\rho\_\{0\}is the initial\-state distribution, andγ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor\. This definition corresponds to a fully specified environment instance with fixed dynamics and rewards\.

##### Partially Observable MDP \(POMDP\)\.

To model partial observability, we consider a partially observable MDP \(POMDP\), defined asℳ=\(𝒮,𝒜,𝒪,P,r,ℐ,ρ0,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{O\},P,r,\\mathcal\{I\},\\rho\_\{0\},\\gamma\), where𝒪\\mathcal\{O\}is the observation space andℐ\(⋅∣s\)\\mathcal\{I\}\(\\cdot\\mid s\)is the observation function mapping latent states to observations\. The underlying environment dynamics\(P,r\)\(P,r\)remain fixed, but the agent only observesot∼ℐ\(⋅∣st\)o\_\{t\}\\sim\\mathcal\{I\}\(\\cdot\\mid s\_\{t\}\)\.

##### Underspecified POMDP\.

Following[Dennis et al\. \(2020\)](https://arxiv.org/html/2608.26469#bib.bib9), we model learning across environments using an*underspecified*POMDP \(UPOMDP\),ℳ=\(𝒮,𝒜,𝒪,Θ,P,r,ℐ,ρ0,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\allowbreak\\mathcal\{A\},\\allowbreak\\mathcal\{O\},\\allowbreak\\Theta,\\allowbreak P,\\allowbreak r,\\allowbreak\\mathcal\{I\},\\allowbreak\\rho\_\{0\},\\allowbreak\\gamma\), whereΘ\\Thetais a set of environment instances\. The transition kernel and reward depend onθ\\theta, written asP:𝒮×𝒜×Θ→Δ⁡\(𝒮\)P:\\mathcal\{S\}\\times\\allowbreak\\mathcal\{A\}\\times\\allowbreak\\Theta\\rightarrow\\Delta\(\\mathcal\{S\}\)andr:𝒮×𝒜×𝒮×Θ→ℝr:\\mathcal\{S\}\\times\\allowbreak\\mathcal\{A\}\\times\\allowbreak\\mathcal\{S\}\\times\\allowbreak\\Theta\\rightarrow\\mathbb\{R\}\. Fixingθ\\thetayields a concrete POMDPℳθ=\(𝒮,𝒜,𝒪,Pθ,rθ,ℐ,ρ0,γ\)\\mathcal\{M\}\_\{\\theta\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{O\},P\_\{\\theta\},r\_\{\\theta\},\\mathcal\{I\},\\rho\_\{0\},\\gamma\)\. We identify an instanceθ∈Θ\\theta\\in\\Thetawith its induced POMDPℳθ\\mathcal\{M\}\_\{\\theta\}\(i\.e\., we useθ\\thetaandℳθ\\mathcal\{M\}\_\{\\theta\}interchangeably in the remainder of the paper\)\.

##### Learning Objective\.

Given an underspecified POMDP characterized by a set of environmentsΘ\\Theta, our learning objective is applying reinforcement learning\([Sutton and Barto, 2018](https://arxiv.org/html/2608.26469#bib.bib24)\)to find a policy that performs optimally across all environments\. LetΠ\\Pidenote the policy class, and letV⁡\(π,θ\):=𝔼ρ0,π,Pθ​\[∑n=0∞γn​rn\]V\(\\pi,\\theta\):=\\mathbb\{E\}\_\{\\rho\_\{0\},\\,\\pi,\\,P^\{\\theta\}\}\\\!\\left\[\\sum\_\{n=0\}^\{\\infty\}\\gamma^\{n\}r\_\{n\}\\right\]denote the expected discounted return of policyπ∈Π\\pi\\in\\Piin environmentℳθ\\mathcal\{M\}\_\{\\theta\}\. The regret of policyπ\\pion environmentθ\\thetais defined as

Regret⁡\(π,θ\):=−V⁡\(π,θ\)\+maxπ′∈Π⁡V⁡\(π′,θ\),\\mathrm\{Regret\}\(\\pi,\\theta\)\\;:=\\;\-V\(\\pi,\\theta\)\\;\+\\;\\max\_\{\\pi^\{\\prime\}\\in\\Pi\}V\(\\pi^\{\\prime\},\\theta\),which measures the suboptimality ofπ\\pirelative to the optimal policy forθ\\theta\.

Our goal is to learn a policyπ⋆∈Π\\pi^\{\\star\}\\in\\Pithat performs well across the entire environment setΘ\\Thetainduced by the UPOMDP\. Ideally, this corresponds to the objectiveπ⋆∈arg⁡minπ∈Π​minθ∈Θ⁡Regret⁡\(π,θ\)\\pi^\{\\star\}\\in\\arg\\min\_\{\\pi\\in\\Pi\}\\allowbreak\\min\_\{\\theta\\in\\Theta\}\\allowbreak\\mathrm\{Regret\}\(\\pi,\\theta\)which captures the goal of achieving low regret over all environment instances\. Throughout the paper, we refer to eliminating regret on an environment instance as*mastering*that environment\.

### 2\.2Curriculum of Environments as a Graph

We make the natural ordering structure exploited by many curriculum methods—discussed in Section[1](https://arxiv.org/html/2608.26469#S1)—explicit by representing the curriculum as a directed acyclic graph \(DAG\) over the environment spaceΘ\\Theta\([Svetlik et al\., 2017](https://arxiv.org/html/2608.26469#bib.bib31);[Narvekar et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib2)\)\.

###### Definition 2\.1\(Curriculum DAG\)\.

A curriculum is a directed acyclic graph𝒢=\(Θ,ℰ\)\\mathcal\{G\}=\(\\Theta,\\mathcal\{E\}\), where each node corresponds to an environment instanceθ∈Θ\\theta\\in\\Theta, and each directed edge\(θ,θ′\)∈ℰ\(\\theta,\\theta^\{\\prime\}\)\\in\\mathcal\{E\}indicates that training onθ\\thetais intended to facilitate learning onθ′\\theta^\{\\prime\}\.

A directed pathP=\(θ1,…,θL\)P=\(\\theta\_\{1\},\\ldots,\\theta\_\{L\}\)in𝒢\\mathcal\{G\}defines a curriculum sequence, where environments are trained in the order specified by the path\. A path is maximal if it starts from a source node \(in\-degree00\) and ends at a sink node \(out\-degree00\)\. Let𝒫⁡\(𝒢\)\\mathcal\{P\}\(\\mathcal\{G\}\)denote the set of all maximal paths in the curriculum DAG\.

##### Curriculum as prerequisite structure\.

Along any maximal pathP=\(θ1,…,θL\)∈𝒫P=\(\\theta\_\{1\},\\ldots,\\theta\_\{L\}\)\\in\\mathcal\{P\}, we assume that respecting the path order enables efficient progression: there exists a constantm\>0m\>0such that for eachθi\\theta\_\{i\}on the path, onceθi−1\\theta\_\{i\-1\}is mastered, allocating at mostmmeffective training interactions onθi\\theta\_\{i\}suffices to learnθi\\theta\_\{i\}This captures the core advantage of structured curricula over unstructured training: prerequisite dependencies reduce the effective cost of mastering harder instances compared to treating environments as independent\.

## 3Proposed Methods

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/overview.png)Figure 1:Overview ofPATH\. \(a\) A environment distributionΘ\\Theta\. \(b\) The same space viewed as an explicit curriculum DAG𝒢\\mathcal\{G\}, where edges encode prerequisite relations between environments\. \(c\)PATH:Randomexplores𝒢\\mathcal\{G\}by sampling random paths \(yellow trajectories\); blue nodes are explicitly visited and learned, while yellow nodes denote additional environments learned via generalization from nearby visited nodes\. \(d\)PATH:Activeidentifies a node with high learning potential, then it allocates more sampling by launching multiple random paths \(purple\) from that node\. \(e\) This adaptive allocation rapidly increases coverage and efficient learns all nodes in𝒢\\mathcal\{G\}\.We propose a new algorithmPATHthat exploits curriculum structure defined over a directed acyclic graph \(DAG\) on the setΘ\\Theta\. The method consists of two components: a randomized path exploration procedure \(PATH:Random\) and a regret\-driven active learning procedure \(PATH:Active\)\.

### 3\.1PATH:Random

PATH:Randomperforms stochastic curriculum exploration by maintainingNNpath pointers on a curriculum DAG𝒢\\mathcal\{G\}over environmentsΘ\\Theta\. Each pointer identifies a current environmentθ∈Θ\\theta\\in\\Theta\. At each iteration, we uniformly sample a batch of pointers and update the student policyπ\\piusing rollouts collected in their current environments\. Equivalently, this is uniform sampling from a dynamic bufferℬ\\mathcal\{B\}of sizeNN, where each entry is the environment currently pointed to by a path\.

Pointers advance via on\-the\-fly path generation, avoiding explicit enumeration of𝒢\\mathcal\{G\}whenever successors can be constructed from the current pointer via local edits or parameter increments\. Each path pointer starts from a uniformly sampled source node\. When a path pointer advances, it samples one outgoing edge uniformly and moves to the successor\. If it reaches a sink, it is restarted from a new uniformly sampled source\. We refer to this path sampling method asrandom path sampling\. Advancement is controlled by two stopping rules\. \(1\)*Early\-stop reward*εes\\varepsilon\_\{\\mathrm\{es\}\}: In each policy update, for every sampled environmentθ\\thetawe collectKθK\_\{\\theta\}episodes with episodic returns\{Rθ,1,…,Rθ,Kθ\}\\\{R\_\{\\theta,1\},\\ldots,R\_\{\\theta,K\_\{\\theta\}\}\\\}, and compute the mean returnR¯θ=1Kθ​∑k=1KθRθ,k\\bar\{R\}\_\{\\theta\}=\\frac\{1\}\{K\_\{\\theta\}\}\\sum\_\{k=1\}^\{K\_\{\\theta\}\}R\_\{\\theta,k\}\. IfR¯θ≥εes\\bar\{R\}\_\{\\theta\}\\geq\\varepsilon\_\{\\mathrm\{es\}\}, we treatθ\\thetaas solved and advance immediately\. \(2\)*Sampling patience*εp\\varepsilon\_\{\\mathrm\{p\}\}: ifθ\\thetahas been sampledεp\\varepsilon\_\{\\mathrm\{p\}\}times without meetingεes\\varepsilon\_\{\\mathrm\{es\}\}, we terminate and reinitialize the path\. In practice,εp\\varepsilon\_\{\\mathrm\{p\}\}is set to approximate the sample budgetmmrequired to efficiently learn anyθ\\thetaalong the curriculum path\. We defer the sensitivity analysis overεes,εp\\varepsilon\_\{\\mathrm\{es\}\},\\varepsilon\_\{\\mathrm\{p\}\}to Appendix[C\.4\.2](https://arxiv.org/html/2608.26469#A3.SS4.SSS2)\.

### 3\.2PATH:Active

PATH:Activereplaces uniform sampling with regret\-weighted replay over the current path pointers\. We maintainNNcurriculum trajectories in parallel and collect their current pointer environments into a dynamic bufferℬ⊂Θ\\mathcal\{B\}\\subset\\Theta\. Eachθ∈ℬ\\theta\\in\\mathcal\{B\}is assigned a nonnegative weightw⁡\(θ\)w\(\\theta\)\(initialized uniformly\)\. At each iteration, we sample a training batch fromℬ\\mathcal\{B\}proportionally tow⁡\(θ\)w\(\\theta\)\. For each sampledθ\\theta, we use rollouts inθ\\thetato collect TD\-error\{δtθ\}\\\{\\delta\_\{t\}^\{\\theta\}\\\}at each timett\. We updatew⁡\(θ\)w\(\\theta\)using the regret proxy–postive value lossused byPLR\([Jiang et al\., 2021b](https://arxiv.org/html/2608.26469#bib.bib10)\):

S⁡\(θ\):=1T​∑t=0Tmax⁡\(∑k=tT\(γ​λ\)k−t​δkθ,0\),S\(\\theta\):=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\}\\max\\\!\\left\(\\sum\_\{k=t\}^\{T\}\(\\gamma\\lambda\)^\{k\-t\}\\delta\_\{k\}^\{\\theta\},\\,0\\right\),whereλ\\lambdaandγ\\gammaare theGeneralized Advantage Estimation\([Schulman et al\., 2016](https://arxiv.org/html/2608.26469#bib.bib25)\)hyperparameters\. HigherS⁡\(θ\)S\(\\theta\)indicates that the policy’s most recent interactions withθ\\thetayielded unexpectedly high returns, suggesting strong learning potential\.

We use the same early\-stop rule asPATH:Random: if the policy’s recent reward onθ\\thetareachesεes\\varepsilon\_\{\\mathrm\{es\}\}, we advance by sampling an outgoing edge fromθ\\thetain𝒢\\mathcal\{G\}\. When this happens, we also expand exploring fromθ\\thetaby inserting a batch of its successor pointers into the bufferℬ\\mathcal\{B\}, so subsequent sampling dispatches multiple paths through the successor region\. All newly advanced pointers are inserted intoℬ\\mathcal\{B\}; to maintain\|ℬ\|=N\|\\mathcal\{B\}\|=N, we evict low\-weight pointers so the buffer stays concentrated on high regret regions while still tracking curriculum progress\. At each iteration, with the replay ratepp, we choose between \(i\) sampling a batch of pointers fromℬ\\mathcal\{B\}, or \(ii\) inserting an initialized random batch of source nodes intoℬ\\mathcal\{B\}\. Injecting new start nodes serves two purposes: \(i\) sampling them leads to revisit explored paths—if a node already meets theεes\\varepsilon\_\{\\mathrm\{es\}\}, we advance immediately without wasting training budget; otherwise, additional updates help the policy relearn it and mitigate catastrophic forgetting\([Rolnick et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib51)\); and \(ii\) new added nodes cause evicting low\-regret pointers, which increases the turnover of paths tracked inℬ\\mathcal\{B\}, speeding up the discovery of high\-regret regions\.

### 3\.3PATH

PATHis a two\-stage algorithm\. We start with the random phase, which efficiently advances along diverse curriculum paths\. Letnndenote the number of maximal curriculum paths that have terminated\. We check the switch conditionn≥Tswitchn\\geq T\_\{\\mathrm\{switch\}\}to ensure a more consistent switch point across runs; once satisfied, we switch toPATH:Active\.

At the transition, we continue training withPATH:Activeon the same buffer\. Intuitively,TswitchT\_\{\\mathrm\{switch\}\}is chosen so that the random stage has already achieved broad exploration; beyond this point, path terminations and revisits become frequent, so reallocating sampling toward high\-learning\-potential regions is more effective\. Figure[1](https://arxiv.org/html/2608.26469#S3.F1)provides an overview ofPATHand its two\-stage training procedure\. We provide one simplified version of PATH in Algorithm[1](https://arxiv.org/html/2608.26469#alg1)\. Full algorithmic details for all three algorithms are provided in Appendix[A](https://arxiv.org/html/2608.26469#A1)\.

Algorithm 1PATH1:Input:curriculum DAG

𝒢\\mathcal\{G\}, pointer buffer size

NN, replay rate

pp, weights

w⁡\(⋅\)w\(\\cdot\), early\-stop threshold

εes\\varepsilon\_\{\\mathrm\{es\}\}, sampling patience

εp\\varepsilon\_\{\\mathrm\{p\}\}, transition time

TswitchT\_\{\\mathrm\{switch\}\}
2:Initialize:policy

π⁡\(ϕ\)\\pi\(\\phi\); fill buffer

ℬ\\mathcal\{B\}with source nodes of

𝒢\\mathcal\{G\}; initialize weights

wwon

ℬ\\mathcal\{B\}
3:whilenot convergeddo

4:if

n≤Tswitchn\\leq T\_\{\\mathrm\{switch\}\}then

5:Sample node

θ∼Unif⁡\(ℬ\)\\theta\\sim\\mathrm\{Unif\}\(\\mathcal\{B\}\)and train

π\\pion

θ\\theta
6:Change

θ\\thetato one of its successor in

𝒢\\mathcal\{G\}if

εes\\varepsilon\_\{\\mathrm\{es\}\}meets, drop it if

εp\\varepsilon\_\{\\mathrm\{p\}\}meets

7:If

θ\\thetadrops, add one source node from

𝒢\\mathcal\{G\}to

ℬ\\mathcal\{B\}
8:

n←n\+1n\\leftarrow n\+1
9:else

10:Sample decision

d∼Bernoulli⁡\(p\)d\\sim\\mathrm\{Bernoulli\}\(p\)
11:if

d=1d=1then

12:Sample node

θ∼ℬ\\theta\\sim\\mathcal\{B\}with probability

∝w⁡\(θ\)\\propto w\(\\theta\)
13:Train

π\\pion sampled

θ\\thetaand update

w⁡\(θ\)←S⁡\(θ\)w\(\\theta\)\\leftarrow S\(\\theta\)
14:Add one

θ\\thetasuccessor in

𝒢\\mathcal\{G\}to

ℬ\\mathcal\{B\}if

εe​s\\varepsilon\_\{es\}meets

15:else

16:Sample a source node of

𝒢\\mathcal\{G\}, add to

ℬ\\mathcal\{B\}
17:Evict low

wwnodes from

ℬ\\mathcal\{B\}to keep

\|ℬ\|=N\|\\mathcal\{B\}\|=N

### 3\.4From Random to Active Path Allocation

We now motivate the necessity of the two phases ofPATH\. We formulate our learning goal as mastering an*unknown*reachable subgraph𝒢¯⊆𝒢\\bar\{\\mathcal\{G\}\}\\subseteq\\mathcal\{G\}induced by the policy’s current capability\.εes\\varepsilon\_\{\\mathrm\{es\}\}determines how far the policy can progress along any curriculum path: once a node fails to reachεes\\varepsilon\_\{\\mathrm\{es\}\}, the successor region beyond that node becomes effectively unreachable at the current stage\. Thus, the leaves of𝒢¯\\bar\{\\mathcal\{G\}\}mark the policy’s current frontier in the environment spaceΘ\\Theta\. We measure progress by*generalization coverage*over𝒢¯\\bar\{\\mathcal\{G\}\}\.

##### Generalization coverage\.

Let𝒢¯=\(V,E\)\\bar\{\\mathcal\{G\}\}=\(V,E\)\. For a maximal curriculum pathPP, letS⁡\(P\)⊆VS\(P\)\\subseteq Vbe its visited nodes\. We use a graph\-local neighborhoodBt​\(u\)=\{v∈V:d⁡\(u,v\)≤t\}B\_\{t\}\(u\)=\\\{v\\in V:d\(u,v\)\\leq t\\\}\(whered⁡\(u,v\)d\(u,v\)is number of edges of the shortest path fromuutovvin undirected version of𝒢¯\\mathcal\{\\bar\{G\}\}\) to represent the maximal region that*could*be influenced by training onuu\. The*realized*generalization is modeled as an unknown subsetG​\(u\)⊆Bt​\(u\)G\(u\)\\subseteq B\_\{t\}\(u\), i\.e\., the set of nodes that become mastered due to generalization when training onuuunder the current policy\. We do not assumeG⁡\(u\)G\(u\)fills the entirett\-ball, as generalization is typically partial and depends on specific nodes\. A path induces coverageC⁡\(P\)=⋃u∈S⁡\(P\)G⁡\(u\)C\(P\)=\\bigcup\_\{u\\in S\(P\)\}G\(u\), and fornnsampled paths𝒫n=\(P1,…,Pn\)\\mathcal\{P\}\_\{n\}=\(P\_\{1\},\\dots,P\_\{n\}\)the global coverage isf⁡\(𝒫n\)=\|⋃i≤nC⁡\(Pi\)\|f\(\\mathcal\{P\}\_\{n\}\)=\\big\|\\bigcup\_\{i\\leq n\}C\(P\_\{i\}\)\\big\|\. Our goal is to expandf⁡\(𝒫n\)f\(\\mathcal\{P\}\_\{n\}\)over𝒢¯\\bar\{\\mathcal\{G\}\}using as few sampled paths as possible\.

##### PATH:Random: growing low\-overlap neighborhoods\.

In the random phase, we sample paths by choosing a child uniformly at each step, inducing a distributionπ\\piover maximal paths\. DrawP1,…,Pn∼i​i​dπP\_\{1\},\\dots,P\_\{n\}\\stackrel\{\{\\scriptstyle iid\}\}\{\{\\sim\}\}\\piand define the marginal gainΔk:=f⁡\(𝒫k\)−f⁡\(𝒫k−1\)\\Delta\_\{k\}:=f\(\\mathcal\{P\}\_\{k\}\)\-f\(\\mathcal\{P\}\_\{k\-1\}\), whereΔk=\|C⁡\(Pk\)∖⋃i<kC⁡\(Pi\)\|\\Delta\_\{k\}=\\big\|C\(P\_\{k\}\)\\setminus\\bigcup\_\{i<k\}C\(P\_\{i\}\)\\big\|\. SinceG​\(u\)⊆Bt​\(u\)G\(u\)\\subseteq B\_\{t\}\(u\), each cover set satisfiesC⁡\(P\)⊆⋃u∈S⁡\(P\)Bt​\(u\)C\(P\)\\subseteq\\bigcup\_\{u\\in S\(P\)\}B\_\{t\}\(u\)\. Therefore, in an early regime where thett\-balls encountered by different random paths have limited overlap, the cover sets\{C⁡\(Pi\)\}\\\{C\(P\_\{i\}\)\\\}also overlap little andΔk\\Delta\_\{k\}remains large, yielding stable growth off⁡\(𝒫n\)f\(\\mathcal\{P\}\_\{n\}\)\. This regime is promoted by the curriculum graph with rich branching and weak bottlenecks, where independent random walks diverge quickly and neighborhood overlap accumulates slowly as more paths are sampled\.

Asnncontinues growing,⋃i≤n⋃u∈S⁡\(Pi\)Bt​\(u\)\\bigcup\_\{i\\leq n\}\\allowbreak\\bigcup\_\{u\\in S\(P\_\{i\}\)\}\\allowbreak B\_\{t\}\(u\)\\allowbreakbegins to saturate: additional random paths increasingly revisit previously covered neighborhoods, so newly visited nodesuutypically haveBt​\(u\)B\_\{t\}\(u\)largely contained in the existing union\. WhenPkP\_\{k\}visits new nodes, they often lie inside the saturated neighborhood and contribute little new mass tof⁡\(𝒫n\)f\(\\mathcal\{P\}\_\{n\}\)\. More importantly, because realized generalization is a subsetG​\(u\)⊆Bt​\(u\)G\(u\)\\subseteq B\_\{t\}\(u\), the remaining unmastered pointsBt​\(u\)∖G​\(u\)B\_\{t\}\(u\)\\setminus G\(u\)can form a small residual inside visitedtt\-balls\. Randomly sampling additional paths then becomes an inefficient way to repeatedly hit this residual\.

##### PATH:Active: allocating budget to the residual area\.

The active phase targets precisely this residual regime\. Using regret signals, we identify under\-mastered nodes with high learning potential and allocate additional budget by dispatching multiple paths from these anchors, rather than continuing uniform exploration over the whole graph\. This exploits prerequisite structure of the DAG curriculum: we assume a high regret node implies that its successor region contains further under\-mastered nodes\. Concentrated sampling increases the chance of repeatedly training on the residual portion ofG⁡\(⋅\)G\(\\cdot\)that random exploration hits only rarely\. However, the same concentration explains weaker*early*coverage for purely active sampling: dispatching many paths from a few starters makes visitedtt\-balls highly overlapping, so the neighborhood grows slowly at the beginning and yields less broad coverage than diversified random paths in the early stage\. Moreover, in the early regime where regret is high for most entries inℬ\\mathcal\{B\}, the fixed replay rate inPATH:Active—inserting newly random source nodes and evicting low\-regret pointers—cannot reliably isolate truly low regret entries\. As a result, it may evict paths that are still high\-regret but under\-trained, prematurely truncating progression and slowing early learning\.

Overall,PATHleverages two complementary mechanisms: \(i\)PATH:Randomrapidly expands the union of \(mostly non\-overlapping\)tt\-balls early, yielding large coverage gains; and \(ii\)PATH:Activelater reallocates training budget to the residual unmastered mass, yielding steadier improvement once random exploration becomes inefficient\. We illustrate the predicted behaviors of the three algorithms in Figure[2](https://arxiv.org/html/2608.26469#S3.F2), and give the empirical evidence to support our analysis in section[4\.3](https://arxiv.org/html/2608.26469#S4.SS3)\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/illustration.png)Figure 2:Conceptual comparison of training budget versus curriculum graph coverage\. Oracle denotes the theoretical optimum\.PATHswitches at updateTT\.

## 4Experiments

We evaluatePATHagainst classical curriculum baselines from unsupervised environment design, focusing on methods that construct curricula solely through sampling of the environment setΘ\\Theta111Our code is available at[https://github\.com/Liu\-Zhenya/PATH](https://github.com/Liu-Zhenya/PATH)\.\. In all experiments, the student agent is trained using Proximal Policy Optimization \(PPO\) \(\([Schulman et al\., 2017](https://arxiv.org/html/2608.26469#bib.bib19)\)\)\. To isolate the effect of curriculum design, we report performance primarily as a function of the number of student policy\-gradient updates \(with each update computed from a fixed number of environment steps\), which is comparable across methods\.

Our baselines include DR\([Tobin et al\., 2017](https://arxiv.org/html/2608.26469#bib.bib6)\), PLR\([Jiang et al\., 2021b](https://arxiv.org/html/2608.26469#bib.bib10);[Jiang et al\., 2021a](https://arxiv.org/html/2608.26469#bib.bib17)\), and ACCEL\([Parker\-Holder et al\., 2022a](https://arxiv.org/html/2608.26469#bib.bib11)\)\. Among them, ACCEL serves as the strongest replay\-based baseline under this setting\. These baselines span increasingly structured sampling: DR uses uniform randomization, PLR performs score\-based replay, and ACCEL/PATHincorporate active selection within curriculum structure\. ACCEL mutates an environment immediately after sampling and does not explicitly enforce progression along a single curriculum path\. In contrast,PATHformulates training as active learning over a curriculum graph, advancing along a path only after the preceding environment meets the mastery criterion\. For hyperparameters and compute statistics, see Section[B\.4](https://arxiv.org/html/2608.26469#A2.SS4)\.

We evaluate on two highly representative RL testbeds that admit general purpose curriculum constructions\. The first one, MiniGrid\([Chevalier\-Boisvert et al\., 2018](https://arxiv.org/html/2608.26469#bib.bib23)\), instantiates sparse\-reward, discrete control, where curricula can be defined as a topologically contiguous sequence of environments obtained by incremental edits \(e\.g\., progressively adding obstacles\), yielding an explicit notion of curriculum progression\. The second evaluated task, BipedalWalker\([Brockman et al\., 2016](https://arxiv.org/html/2608.26469#bib.bib21)\), instantiates dense\-reward, continuous control, where curricula are most naturally specified in an environment\-parameter space by monotonically increasing difficulty \(e\.g\., harder terrain configurations\)\. These benchmarks cover broad classes of practical RL problems, and the curriculum constructions we use—topologically contiguous edits and monotone difficulty schedules—are generic templates that apply to real world applications\. For each environment, we evaluate both robustness and generalization, using metrics such as IQM \(interquartile mean\)\([Agarwal et al\., 2021](https://arxiv.org/html/2608.26469#bib.bib41)\), mean return, and solved rate\. We additionally evaluatePATHon Leaper from Procgen\([Cobbe et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib33)\), an implicit\-parameterization benchmark where difficulty is inferred from observable environmental axes rather than explicit parameters \(Appendix[C\.3](https://arxiv.org/html/2608.26469#A3.SS3)\)\.

### 4\.1MiniGrid

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/three_seeds_combined.png)Figure 3:Examples of MiniGrid layouts from generalization test\-set thatPATHsolves\.We first consider a partially observable navigation task\([Dennis et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib9)\)based on MiniGrid\. The environment setΘ\\Thetaconsists of all15×1515\\times 15maze layouts obtained by placing between00and6060blocks in an initially empty room\. The agent starts in one location and aim to find the goal\. We define a curriculum DAG overΘ\\Theta, in which each node corresponds to a particular room layout and each directed edge denotes an edit that adds blocks\. For a fair comparison, we mirror the implicit curriculum induced by ACCEL’s editing procedure\([Parker\-Holder et al\., 2022a](https://arxiv.org/html/2608.26469#bib.bib11)\): each curriculum path begins from a randomly generated empty room, and each transition applies a single edit that adds a random number of blocks uniformly in \[1,5\]\. A path terminates once the layout reaches the maximum complexity of 60 blocks\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/mg_complexity.png)Figure 4:Left: emergent complexity of sampled MiniGrid layouts during training\. Right: average episodic return on layouts sampled during training\. Curves show mean±\\pmstandard error across 5 training seeds\.Full environment and curriculum specifications are provided in Appendix[B\.1](https://arxiv.org/html/2608.26469#A2.SS1)\. Agents are trained for2020k PPO updates using an LSTM\-based policy operating on partially observable inputs\. We find that, up to2020k updates,PATHremains inPATH:Randomand continues to make steady progress, with no evidence of stagnation\. In Figure[4](https://arxiv.org/html/2608.26469#S4.F4), we report the sampling\-complexity metric alongside mean training returns\.PATHsamples complex layouts stably and reaches a higher training return in fewer updates than all other methods\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/mg_robust_visualization.png)Figure 5:Aggregate robustness performance \(solved rates\) in MiniGrid\. The vertical line indicates the mean, and the colored bar shows±1\\pm 1standard error\.![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_training_curves.png)Figure 6:Performance on BipedalWalker robustness tests during training \(mean±\\pmstandard error\), evaluated every11k PPO updates\.We assess robustness by evaluating the solved rate by each method on ten hand\-crafted challenging environments\. The complete set of robustness test layouts and per\-environment results are reported in Figure[12](https://arxiv.org/html/2608.26469#A3.F12)and Table[4](https://arxiv.org/html/2608.26469#A3.T4)in appendix[C\.1\.1](https://arxiv.org/html/2608.26469#A3.SS1.SSS1)\.PATHsolves the majority of these test environments As summarized in the aggregate robustness statistics \(Figure[5](https://arxiv.org/html/2608.26469#S4.F5)\), while ACCEL also achieves near\-perfect IQM,PATHattains a mean score of up to98%98\\%, whereas ACCEL and PLR are around80%80\\%\.

Table 1:Generalization test solved rates at2020k updates \(mean±\\pmstandard error\) for55runs of each method on MiniGrid\. Bold values are within one standard error of the best mean\.We further evaluate generalization by sampling10001000random layouts fromΘ\\Theta\. Table[1](https://arxiv.org/html/2608.26469#S4.T1)shows that all methods exceed95%95\\%solved rate, withPATHachieving a near\-perfect99\.4%99\.4\\%solved rate, indicating strong generalization\. Crucially,PATHperforms well on both robustness and generalization, which we attribute to its broad coverage of the environment space\. Figure[3](https://arxiv.org/html/2608.26469#S4.F3)shows representative layouts that are solved byPATHafter2020k updates; additional details of the generalization evaluation are provided in Appendix[C\.1\.2](https://arxiv.org/html/2608.26469#A3.SS1.SSS2)\. Notably, the random exploration stage already reaches near\-perfect performance across the space, which we attribute to broad coverage from diverse random paths coupled with curriculum progression\.

### 4\.2BipedalWalker

We next evaluate on the modified version of BipedalWalker\([Wang et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib22)\)\. Following the definition from[Parker\-Holder et al\. \(2022a\)](https://arxiv.org/html/2608.26469#bib.bib11), each environment is parameterized by a 8\-dimensional vectorθ\\thetathat specifies terrain attributes \(e\.g\., gaps, stumps, stairs, and surface roughness\), so the environment family induces a continuous, multi\-dimensional spaceΘ\\Theta\. We discretizeΘ\\Thetafrom a designated easy start configuration under the same editor used by ACCEL\. Concretely, each curriculum path begins at a simple terrain and advances complexity such as adding stumps, stairs by incrementing one or more coordinates ofθ\\theta\. Full details of the parameterization, discretization, and curriculum construction of BipedalWalker are provided in Appendix[C\.2](https://arxiv.org/html/2608.26469#A3.SS2)\.

Similar to MiniGrid, each algorithm is evaluated at2020k PPO updates using robustness and generalization metrics\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_robust_visualization.png)Figure 7:Aggregate normalized robustness performance \(returns\) in BipedalWalker\. Returns are min–max normalized from\[−100,300\]\[\-100,300\]to\[0,1\]\[0,1\]\. The vertical line indicates the mean, and the colored bar shows±1\\pm 1standard error\.To evaluate robustness, we adopt the held\-out test environment classes from[Parker\-Holder et al\. \(2022a\)](https://arxiv.org/html/2608.26469#bib.bib11):Hardcore,Roughness,Stairs,PitGap, andStump\. We evaluate 128 fixed environments per class for each method, and examples of each class are given at Figure[14](https://arxiv.org/html/2608.26469#A3.F14)in Appendix[C\.2\.1](https://arxiv.org/html/2608.26469#A3.SS2.SSS1)\. Figure[6](https://arxiv.org/html/2608.26469#S4.F6)reports per\-class performance over training, and Figure[7](https://arxiv.org/html/2608.26469#S4.F7)summarizes aggregate normalized\-return statistics at the2020k checkpoint across classes\. Overall,PATHconsistently outperforms the baselines on every robustness class, reaching approximately 90% in both IQM and mean normalized return at2020k updates, whereas ACCEL remains below 60%\. Full class\-wise results at2020k are reported in Table[5](https://arxiv.org/html/2608.26469#A3.T5)in Appendix[C\.2\.1](https://arxiv.org/html/2608.26469#A3.SS2.SSS1)\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_radar.png)Figure 8:Radar plots of terrain parameters for all solved BipedalWalker terrains from a single run of each algorithm\. Axes are ordered clockwise starting at 12 o’clock:Roughness\(max 10\),Gap\(max 10\),Stump\(max 5\),Stairs\(max 5\), andSteps\(max 9\)\. Each polyline corresponds to one solved terrain\.To test generalization, we uniformly sample10001000configurations fromΘ\\Thetato form a fixed test set \(Appendix[C\.2\.2](https://arxiv.org/html/2608.26469#A3.SS2.SSS2)\), and report each method’s solved rate in Table[2](https://arxiv.org/html/2608.26469#S4.T2)\. Although most configurations inΘ\\Thetaare unsolvable for the trained policy,PATHachieves substantially higher average test returns \(≥40\\geq 40\) and solved rates \(≥10%\\geq 10\\%\) than all baselines\. To further characterize where each method succeeds, we collect the set of test configurations it solves and visualize them in Figure[8](https://arxiv.org/html/2608.26469#S4.F8)\. We seePATHexhibits markedly broader and denser coverage of the environment space\. Beyond better generalization,PATHlocates the policy’s learning frontier inΘ\\Theta, thereby identifying the region that is learnable in practice\. Figure[9](https://arxiv.org/html/2608.26469#S4.F9)shows a representative test environment thatPATHsolves while other methods fail after2020k updates\. Together, these results indicate thatPATHis more generalized and more robust than competing approaches\. Overall, Figures[4](https://arxiv.org/html/2608.26469#S4.F4)and[6](https://arxiv.org/html/2608.26469#S4.F6)show clear advantages for methods that exploit the curriculum graph structure \(PATH,ACCEL\) over baselines that do not \(PLR,DR\)\. This supports the prerequisite property in Section[2\.2](https://arxiv.org/html/2608.26469#S2.SS2): leveraging curriculum structure yields a consistent training benefit\.PATHfurther amplifies this advantage by actively refining path selection over the graph, outperformingACCEL\.

Table 2:Generalization test returns and solved rates at2020k updates \(mean±\\pmstandard error\) for55runs of each method on BipedalWalker\. Bold values are within one standard error of the best mean\.![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_solved.png)Figure 9:Examples of BipedalWalker terrains from the generalization test thatPATHsolves\.
### 4\.3Random Exploration vs\. Active Path Allocation

For BipedalWalker we observe thatPATHswitches at88k updates when a sufficient number of sampled paths ended\. To disentangle the effect of this phase transition, we take the88k checkpoint ofPATHand continue training it to2020k while keeping applyingPATH:Random\. In addition, we include aPATH:Activebaseline trained for the full2020k updates\. This setup enables a comprehensive comparison amongPATHwith its two components\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_pitgap_ablation.png)\(a\)First\.
![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/test_stump_comparison.png)\(b\)Second\.

Figure 10:Ablation results onPitGap\. \(a\) Robustness test returns versus PPO updates forPATH,PATH:Active, andPATH:Random\(phase shift at∼\\sim88k updates\)\. \(b\) Corresponding sampling complexity over dimension ofPitGapversus PPO updates for33methods\. Both figures show the mean±\\pmstandard error\. PATH overlaps withPATH:RANDOMin the first88k updates\.![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_general_bar.png)Figure 11:Bar figures of generalization test returns and solved rates at2020k updates \(mean±\\pmstandard error\) for55runs of PATH and its two components\. bar height is the mean and segment shows one standard error\.We follow the robustness and generalization tests in Section[4\.2](https://arxiv.org/html/2608.26469#S4.SS2)\. Robustness results are summarized in Figure[17](https://arxiv.org/html/2608.26469#A3.F17)in Appendix[C\.4\.1](https://arxiv.org/html/2608.26469#A3.SS4.SSS1)and one case study is shown in Figure[10](https://arxiv.org/html/2608.26469#S4.F10)\. In Figure[10\(a\)](https://arxiv.org/html/2608.26469#S4.F10.sf1),PATH:Randomperforms stronger early but then plateaus, whereasPATH:Activeimproves more slowly at first but continues improvements up to2020k updates, eventually overtakingPATH:Random;PATHachieves superior performance in both start and end of training\. Figure[10\(b\)](https://arxiv.org/html/2608.26469#S4.F10.sf2)visualizes the correspondingPitGapdimension of the parameters on sampled environments during training\.PATH:Randomrapidly increases sampling complexity early, but later exhibits high repetition with limited further progress\. In contrast,PATH:Activesteadily increases sampling complexity throughout training, reaching harder environments by the end\.PATHcombines both behaviors: it explores quickly in the initial random phase and then reallocates sampling toward the most challenging environments, yielding consistently high sampling complexity and the strongest returns\. Figure[10](https://arxiv.org/html/2608.26469#S4.F10)offers a sufficient example that substantiates the discussion in Figure[2](https://arxiv.org/html/2608.26469#S3.F2)\. We provide more direct evidence by evaluating generalization, a metric that more faithfully captures the degree of coverage\. As shown in Figure[11](https://arxiv.org/html/2608.26469#S4.F11),PATHexhibits a clear advantage over bothPATH:RandomandPATH:Active, suggesting that the initial exploration followed by active path selection greatly improves generalization\. Another advantage of combining the two phases is computational: althoughPATH:Activeattains comparable robustness performance in the end, each update iteration takes nearly twice as long as in the random stage \(Appendix[B\.3](https://arxiv.org/html/2608.26469#A2.SS3)\)\. Thus,PATHimproves not only training effectiveness but also wall\-clock efficiency, benefiting from the faster updates enabled by random path sampling\. Collectively, our ablations suggest thatPATHuses the training budget efficiently, achieving broader coverage of the environment space by leveraging both components\.

##### Remarks

We assume access to a curriculum prior and rely on hyperparameters such as the early\-stop thresholdεes\\varepsilon\_\{\\mathrm\{es\}\}\. However, structured curricula are common in challenging RL and robotics tasks, where task difficulty is often increased via manual, multi\-dimensional features’ modifications\([Margolis et al\., 2022](https://arxiv.org/html/2608.26469#bib.bib55);[Atanassov et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib52)\)\. Within this framework, early\-stop serves as a lightweight mastery check for progression rather than a finely tuned control policy\. Our ablations further show thatPATHis robust to hyperparameter choices \(Appendix[C\.4\.2](https://arxiv.org/html/2608.26469#A3.SS4.SSS2)\)\. We additionally verify PATH’s robustness to noisy DAG structure in Appendix[C\.4\.3](https://arxiv.org/html/2608.26469#A3.SS4.SSS3)\.

## 5Related Work

##### Automated curriculum design for RL\.

Curriculum learning accelerates RL by organizing experience into progressions of difficulty\([Narvekar et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib2);[Parker\-Holder et al\., 2022b](https://arxiv.org/html/2608.26469#bib.bib29);[Soviany et al\., 2022](https://arxiv.org/html/2608.26469#bib.bib30)\)\. A broad literature automates this process via \(i\) teacher–student task selection driven by learning\-progress/competence signals\([Matiisen et al\., 2017](https://arxiv.org/html/2608.26469#bib.bib4);[Portelas et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib14)\), \(ii\) self\-paced reweighting of the training distribution toward harder tasks as the agent improves\([Klink et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib15);[Klink et al\., 2021](https://arxiv.org/html/2608.26469#bib.bib16)\), and \(iii\) goal generation curricula that expand outward from solvable goals to harder instances\([Florensa et al\., 2018](https://arxiv.org/html/2608.26469#bib.bib49);[Colas et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib48)\)\. Some methods generate diverse tasks specifically to induce generalizable skill repertoires for downstream transfer\([Fang et al\., 2021](https://arxiv.org/html/2608.26469#bib.bib42)\)\. Complementary work improves sim\-to\-real transfer by training over randomized environment distributions and focusing sampling on the most useful regions\([Tobin et al\., 2017](https://arxiv.org/html/2608.26469#bib.bib6);[Peng et al\., 2018](https://arxiv.org/html/2608.26469#bib.bib7);[Muratore et al\., 2021](https://arxiv.org/html/2608.26469#bib.bib8);[Mehta et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib37)\), and by injecting additional structure or priors to guide progression\([Tzannetos et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib38);[Li et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib39);[Wang et al\., 2025](https://arxiv.org/html/2608.26469#bib.bib40);[Hsiao et al\., 2025](https://arxiv.org/html/2608.26469#bib.bib36)\)\. Among these autonomous algorithms,[Svetlik et al\. \(2017\)](https://arxiv.org/html/2608.26469#bib.bib31)is closely related to our work in explicitly modeling curricula as a DAG\. However, Svetlik et al\. address a fundamentally different problem: given tasks with*unknown*prerequisite structure, they infer the curriculum DAG from pairwise task transfer matrices—their contribution answers*“what DAG to use?”*Our contribution answers the complementary question:*“given a DAG, how to traverse it efficiently?”*The two approaches are orthogonal and directly composable: Svetlik’s DAG\-inference procedure could construct the input toPATHfor domains without explicit parameterization, creating a full pipeline from unstructured task sets to efficient curriculum traversal\.

##### Unsupervised Environment Design\.

One prominent line of autonomous curriculum learning is Unsupervised Environment Design \(UED\)\([Dennis et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib9)\), which casts training as a bilevel game over minimax regret objective\. A complementary, replay\-based thread approximates this objective by actively selecting and replaying environments from a buffer, with Prioritized Level Replay \(PLR\)\([Jiang et al\., 2021b](https://arxiv.org/html/2608.26469#bib.bib10)\)as a representative approach\.

Recent work has sought to stabilize UED training through better\-motivated objectives, estimators, and optimization procedures\([Jiang et al\., 2021a](https://arxiv.org/html/2608.26469#bib.bib17);[Mediratta et al\., 2023](https://arxiv.org/html/2608.26469#bib.bib43);[Beukman et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib35);[Rutherford et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib13);[Monette et al\., 2025](https://arxiv.org/html/2608.26469#bib.bib44)\)\. Another line improves robustness by encouraging smoother and safer alignment between the generator\-induced environment distribution and a target distribution\([Jiang et al\., 2022](https://arxiv.org/html/2608.26469#bib.bib45);[Azad et al\., 2023](https://arxiv.org/html/2608.26469#bib.bib47);[Costales and Nikolaidis, 2024](https://arxiv.org/html/2608.26469#bib.bib18);[Lee et al\., 2025](https://arxiv.org/html/2608.26469#bib.bib12)\)\. To improve the quality and diversity of the generated environments, some methods refine the regret/priority signals used for selection and replay\([Li et al\., 2023](https://arxiv.org/html/2608.26469#bib.bib46)\), while others incorporate explicit generative models to synthesize environment instances\([Teoh et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib34);[Chung et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib20)\)\. These works improve UED curricula by improving the generator—i\.e\., learning to produce higher\-quality training environments—whereas we assume a fixed pool of environments and study how to learn effectively by sampling within this pool\.

The most closely related work is ACCEL\([Parker\-Holder et al\., 2022a](https://arxiv.org/html/2608.26469#bib.bib11)\), which brings evolutionary ideas from POET\([Wang et al\., 2019](https://arxiv.org/html/2608.26469#bib.bib22)\)with PLR’s prioritization, achieving SOTA performance among replay\-based methods\. In contrast to ACCEL and prior UED approaches that primarily improve how environments are generated or prioritized, our work focuses on exploiting curriculum structure itself: we make the implicit progression induced by evolutions explicit as a curriculum DAG, and introduce a path\-level selection mechanism\.

## 6Conclusion

We introducedPATH, a new curriculum framework that operates active learning over the curriculum itself\. We provide motivation for the two\-phases learning benefiting from rapid exploration followed by regret based concentration\.PATHassumes access to a constructible curriculum DAG, which is natural in parameterized or observable settings but requires additional effort in fully underspecified domains; automatic DAG construction \(e\.g\., via LLMs as in Eurekaverse\([Liang et al\., 2024](https://arxiv.org/html/2608.26469#bib.bib54)\)\) is a promising future direction\. Our evaluation focuses on UED\-family baselines; comparison with a broader set of curriculum learning approaches remains an open direction\. ApplyingPATHto more complex real\-world settings, including 3D embodied environments and sim\-to\-real transfer curricula, is an important next step\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning by improving curriculum learning for reinforcement learning via active path selection on structured curriculum graphs\. The primary anticipated impact is methodological: better sample efficiency and stronger generalization may reduce the interaction and computation needed to train robust policies\. At the same time, more efficient training methods can also accelerate the development and deployment of capable RL agents, which may be applied in domains with safety or misuse concerns\. Our work does not introduce new sensing, surveillance, or data collection mechanisms, and we evaluate only in standard simulated benchmarks; nonetheless, responsible downstream deployment should include domain\-specific safety constraints, testing, and monitoring\.

## Acknowledgements

This work used Bridges\-2 at Pittsburgh Supercomputing Center through allocation CIS240925 from the Advanced Cyberinfrastructure Coordination Ecosystem\([Boerner et al\., 2023](https://arxiv.org/html/2608.26469#bib.bib56)\): Services & Support \(ACCESS\) program, supported by the U\.S\. National Science Foundation \(NSF\) under grants 2138259, 2138286, 2138307, 2137603, and 2138296\. We gratefully acknowledge the support of NSF IIS\-2313131, IIS\-2332475, IIS\-2543755, the US Department of Energy award DE\-EE0009505, the NSF\-Simons AI\-Institute for the Sky \(SkAI\) via grants NSF AST\-2421845 and Simons Foundation MPS\-AI00010513, and the University of Chicago’s Research Computing Center for their support of this work\.

## References

- Agarwalet al\.\(2021\)R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. BellemareDeep reinforcement learning at the edge of the statistical precipice\.InProceedings of the 35th International Conference on Neural Information Processing Systems,NIPS ’21,Red Hook, NY, USA\.External Links:ISBN 9781713845393Cited by:[§4](https://arxiv.org/html/2608.26469#S4.p3.1)\.
- Atanassovet al\.\(2024\)V\. Atanassov, J\. Ding, J\. Kober, I\. Havoutis, and C\. D\. SantinaCurriculum\-based reinforcement learning for quadrupedal jumping: a reference\-free design\.External Links:2401\.16337,[Link](https://arxiv.org/abs/2401.16337)Cited by:[§4\.3](https://arxiv.org/html/2608.26469#S4.SS3.SSS0.Px1.p1.1)\.
- Azadet al\.\(2023\)A\. S\. Azad, I\. Gur, J\. Emhoff, N\. Alexis, A\. Faust, P\. Abbeel, and I\. StoicaCLUTR: curriculum learning via unsupervised task representation learning\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 1361–1395\.External Links:[Link](https://proceedings.mlr.press/v202/azad23a.html)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Bengioet al\.\(2009\)Y\. Bengio, J\. Louradour, R\. Collobert, and J\. WestonCurriculum learning\.InProceedings of the 26th International Conference on Machine Learning,pp\. 41–48\.External Links:[Document](https://dx.doi.org/10.1145/1553374.1553380)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1)\.
- Beukmanet al\.\(2024\)M\. Beukman, S\. Coward, M\. Matthews, M\. Fellows, M\. Jiang, M\. Dennis, and J\. FoersterRefining minimax regret for unsupervised environment design\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Boerneret al\.\(2023\)T\. J\. Boerner, S\. Deems, T\. R\. Furlani, S\. L\. Knuth, and J\. TownsACCESS: advancing innovation: NSF’s advanced cyberinfrastructure coordination ecosystem: services & support\.InPractice and Experience in Advanced Research Computing,PEARC ’23,New York, NY, USA,pp\. 1–4\.External Links:[Document](https://dx.doi.org/10.1145/3569951.3597559),[Link](https://doi.org/10.1145/3569951.3597559)Cited by:[Acknowledgements](https://arxiv.org/html/2608.26469#Sx2.p1.1)\.
- Brockmanet al\.\(2016\)G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. ZarembaOpenAI gym\.InarXiv preprint arXiv:1606\.01540,Cited by:[§B\.2](https://arxiv.org/html/2608.26469#A2.SS2.SSS0.Px1.p1.1),[§C\.2\.1](https://arxiv.org/html/2608.26469#A3.SS2.SSS1.p1.1),[§4](https://arxiv.org/html/2608.26469#S4.p3.1)\.
- Chevalier\-Boisvertet al\.\(2018\)M\. Chevalier\-Boisvert, L\. Willems, and S\. PalMinimalistic gridworld environment for OpenAI gym\.External Links:[Link](https://github.com/Farama-Foundation/Minigrid)Cited by:[§B\.1](https://arxiv.org/html/2608.26469#A2.SS1.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.26469#S4.p3.1)\.
- Chunget al\.\(2024\)H\. Chung, J\. Lee, M\. Kim, D\. Kim, and S\. OhAdversarial environment design via regret\-guided diffusion models\.InAdvances in Neural Information Processing Systems \(NeurIPS\) 2024,Note:SpotlightExternal Links:[Document](https://dx.doi.org/10.5555/3737916.3739951)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Cobbeet al\.\(2020\)K\. Cobbe, C\. Hesse, J\. Hilton, and J\. SchulmanLeveraging procedural generation to benchmark reinforcement learning\.InProceedings of the 37th International Conference on Machine Learning,H\. D\. III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 2048–2056\.External Links:[Link](https://proceedings.mlr.press/v119/cobbe20a.html)Cited by:[§C\.3\.1](https://arxiv.org/html/2608.26469#A3.SS3.SSS1.p1.1),[§4](https://arxiv.org/html/2608.26469#S4.p3.1)\.
- Cobbeet al\.\(2019\)K\. Cobbe, O\. Klimov, C\. Hesse, T\. Kim, and J\. SchulmanQuantifying generalization in reinforcement learning\.External Links:1812\.02341,[Link](https://arxiv.org/abs/1812.02341)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1)\.
- Colaset al\.\(2019\)C\. Colas, P\. Fournier, M\. Chetouani, O\. Sigaud, and P\. OudeyerCURIOUS: intrinsically motivated modular multi\-goal reinforcement learning\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 1331–1340\.External Links:[Link](https://proceedings.mlr.press/v97/colas19a.html)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Costales and Nikolaidis \(2024\)R\. Costales and S\. NikolaidisEnabling adaptive agent training in open\-ended simulators by targeting diversity\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Denniset al\.\(2020\)M\. Dennis, N\. Jaques, E\. Vinitsky, A\. Bayen, S\. Russell, A\. Critch, and S\. LevineEmergent complexity and zero\-shot transfer via unsupervised environment design\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 13049–13061\.Cited by:[§B\.1](https://arxiv.org/html/2608.26469#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.26469#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26469#S2.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.26469#S4.SS1.p1.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p1.1)\.
- Fanget al\.\(2021\)K\. Fang, Y\. Zhu, S\. Savarese, and L\. Fei\-FeiDiscovering generalizable skills via automated generation of diverse tasks\.Robotics: Science and Systems \(RSS\)\.Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Florensaet al\.\(2018\)C\. Florensa, D\. Held, X\. Geng, and P\. AbbeelAutomatic goal generation for reinforcement learning agents\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1515–1528\.External Links:[Link](https://proceedings.mlr.press/v80/florensa18a.html)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Florensaet al\.\(2017\)C\. Florensa, D\. Held, M\. Wulfmeier, M\. Zhang, and P\. AbbeelReverse curriculum generation for reinforcement learning\.InProceedings of the 1st Annual Conference on Robot Learning,S\. Levine, V\. Vanhoucke, and K\. Goldberg \(Eds\.\),Proceedings of Machine Learning Research, Vol\.78,pp\. 482–495\.External Links:[Link](https://proceedings.mlr.press/v78/florensa17a.html)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1)\.
- Hsiaoet al\.\(2025\)V\. Hsiao, M\. Roberts, L\. M\. Hiatt, G\. Konidaris, and D\. S\. NauAutomating curriculum learning for reinforcement learning using a skill\-based bayesian network\.InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems,AAMAS ’25,Richland, SC,pp\. 988–996\.External Links:ISBN 9798400714269Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Jianget al\.\(2021a\)M\. Jiang, M\. D\. Dennis, J\. Parker\-Holder, J\. N\. Foerster, E\. Grefenstette, and T\. RocktäschelReplay\-guided adversarial environment design\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=5UZ-AcwFDKJ)Cited by:[§C\.1\.1](https://arxiv.org/html/2608.26469#A3.SS1.SSS1.p1.1),[§4](https://arxiv.org/html/2608.26469#S4.p2.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Jianget al\.\(2022\)M\. Jiang, M\. Dennis, J\. Parker\-Holder, A\. Lupu, H\. Küttler, E\. Grefenstette, T\. Rocktäschel, and J\. FoersterGrounding aleatoric uncertainty for unsupervised environment design\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 32868–32881\.Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Jianget al\.\(2021b\)M\. Jiang, E\. Grefenstette, and T\. RocktäschelPrioritized level replay\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 4940–4950\.External Links:[Link](https://proceedings.mlr.press/v139/jiang21b.html)Cited by:[§3\.2](https://arxiv.org/html/2608.26469#S3.SS2.p1.1),[§4](https://arxiv.org/html/2608.26469#S4.p2.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p1.1)\.
- Klinket al\.\(2021\)P\. Klink, H\. Abdulsamad, B\. Belousov, C\. D’Eramo, J\. Peters, and J\. PajarinenA probabilistic interpretation of self\-paced learning with applications to reinforcement learning\.J\. Mach\. Learn\. Res\.22\(1\)\.External Links:ISSN 1532\-4435Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Klinket al\.\(2020\)P\. Klink, C\. DEramo, J\. R\. Peters, and J\. PajarinenSelf\-paced deep reinforcement learning\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 9216–9227\.Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Leeet al\.\(2025\)T\. E\. Lee, N\. R\. Ke, S\. Patil, A\. Dahmani, E\. Yiu, E\. Saleh, A\. Gopnik, O\. Kroemer, and G\. BersethExploration by exploitation: curriculum learning for reinforcement learning agents through competence\-based curriculum policy search\.InThe Exploration in AI Today Workshop at ICML 2025,External Links:[Link](https://openreview.net/forum?id=mAeQuz0gbR)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p2.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Liet al\.\(2024\)M\. Li, J\. Zhang, and E\. BareinboimCausally aligned curriculum learning\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hp4yOjhwTs)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2020\)R\. Li, A\. Jabri, T\. Darrell, and P\. AgrawalTowards practical multi\-object manipulation using relational reinforcement learning\.In2020 ieee international conference on robotics and automation \(icra\),pp\. 4051–4058\.Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p2.1)\.
- Liet al\.\(2023\)W\. Li, P\. Varakantham, and D\. LiGeneralization through diversity: Improving unsupervised environment design\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence, IJCAI\-23,E\. Elkind \(Ed\.\),pp\. 5411–5419\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2023/601)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Lianget al\.\(2024\)W\. Liang, Y\. Mu, H\. Tang, G\. Oxholm, E\. Childs, and Y\. ZhuEurekaverse: environment curriculum generation via large language models\.InConference on Robot Learning,Cited by:[§6](https://arxiv.org/html/2608.26469#S6.p1.1)\.
- Margoliset al\.\(2022\)G\. Margolis, G\. Yang, K\. Paigwar, T\. Chen, and P\. AgrawalRapid locomotion via reinforcement learning\.InRobotics: Science and Systems,Cited by:[§4\.3](https://arxiv.org/html/2608.26469#S4.SS3.SSS0.Px1.p1.1)\.
- Matiisenet al\.\(2017\)T\. Matiisen, A\. Oliver, T\. Cohen, and J\. SchulmanTeacher–student curriculum learning\.arXiv preprint arXiv:1707\.00183\.Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Medirattaet al\.\(2023\)I\. Mediratta, M\. Jiang, J\. Parker\-Holder, M\. Dennis, E\. Vinitsky, and T\. RocktäschelStabilizing unsupervised environment design with a learned adversary\.InProceedings of The 2nd Conference on Lifelong Learning Agents,S\. Chandar, R\. Pascanu, H\. Sedghi, and D\. Precup \(Eds\.\),Proceedings of Machine Learning Research, Vol\.232,pp\. 270–291\.External Links:[Link](https://proceedings.mlr.press/v232/mediratta23a.html)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Mehtaet al\.\(2020\)B\. Mehta, M\. Diaz, F\. Golemo, C\. J\. Pal, and L\. PaullActive domain randomization\.InProceedings of the Conference on Robot Learning,L\. P\. Kaelbling, D\. Kragic, and K\. Sugiura \(Eds\.\),Proceedings of Machine Learning Research, Vol\.100,pp\. 1162–1176\.External Links:[Link](https://proceedings.mlr.press/v100/mehta20a.html)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Monetteet al\.\(2025\)N\. Monette, A\. Letcher, M\. Beukman, M\. T\. Jackson, A\. Rutherford, A\. D\. Goldie, and J\. N\. FoersterAn optimisation framework for unsupervised environment design\.External Links:2505\.20659,[Link](https://arxiv.org/abs/2505.20659)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Muratoreet al\.\(2021\)F\. Muratore, C\. Eilers, M\. Gienger, and J\. PetersData\-efficient domain randomization with bayesian optimization\.IEEE Robotics and Automation Letters6\(2\),pp\. 911–918\.External Links:[Document](https://dx.doi.org/10.1109/LRA.2021.3052391)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Narvekaret al\.\(2020\)S\. Narvekar, B\. Peng, M\. Leonetti, J\. Sinapov, M\. E\. Taylor, and P\. StoneCurriculum learning for reinforcement learning domains: a framework and survey\.Journal of Machine Learning Research21\(181\),pp\. 1–50\.External Links:[Link](https://jmlr.org/papers/v21/20-212.html)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.26469#S2.SS2.p1.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1)\.
- Parker\-Holderet al\.\(2022a\)J\. Parker\-Holder, M\. Jiang, M\. Dennis, M\. Samvelyan, J\. Foerster, E\. Grefenstette, and T\. RocktäschelEvolving curricula with regret\-based environment design\.InProceedings of the 39th International Conference on Machine Learning,Cited by:[§B\.2](https://arxiv.org/html/2608.26469#A2.SS2.SSS0.Px1.p1.1),[§B\.4](https://arxiv.org/html/2608.26469#A2.SS4.p1.1),[§C\.2\.1](https://arxiv.org/html/2608.26469#A3.SS2.SSS1.p1.1),[§1](https://arxiv.org/html/2608.26469#S1.p1.1),[§1](https://arxiv.org/html/2608.26469#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.26469#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.26469#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.26469#S4.SS2.p3.1),[§4](https://arxiv.org/html/2608.26469#S4.p2.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p3.1)\.
- Parker\-Holderet al\.\(2022b\)J\. Parker\-Holder, R\. Rajan, X\. Song, A\. Biedenkapp, Y\. Miao, T\. Eimer, B\. Zhang, V\. Nguyen, R\. Calandra, A\. Faust, F\. Hutter, and M\. LindauerAutomated Reinforcement Learning \(AutoRL\): A Survey and Open Problems\.Journal of Artificial Intelligence Research74,pp\. 517–568\.External Links:2201\.03916,ISSN 1076\-9757,[Document](https://dx.doi.org/10.1613/jair.1.13596)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Penget al\.\(2018\)X\. B\. Peng, M\. Andrychowicz, W\. Zaremba, and P\. AbbeelSim\-to\-real transfer of robotic control with dynamics randomization\.In2018 IEEE International Conference on Robotics and Automation \(ICRA\),Vol\.,pp\. 3803–3810\.External Links:[Document](https://dx.doi.org/10.1109/ICRA.2018.8460528)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Portelaset al\.\(2019\)R\. Portelas, C\. Colas, K\. Hofmann, and P\. OudeyerTeacher algorithms for curriculum learning of deep rl in continuously parameterized environments\.In3rd Conference on Robot Learning \(CoRL\),pp\. 835–853\.External Links:[Link](https://proceedings.mlr.press/v100/portelas20a/portelas20a.pdf)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Rolnicket al\.\(2019\)D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. WayneExperience replay for continual learning\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. dAlché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32\.Cited by:[§3\.2](https://arxiv.org/html/2608.26469#S3.SS2.p2.1)\.
- Rutherfordet al\.\(2024\)A\. Rutherford, M\. Beukman, T\. Willi, B\. Lacerda, N\. Hawes, and J\. FoersterNo regrets: investigating and improving regret approximations for curriculum discovery\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Schulmanet al\.\(2016\)J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. AbbeelHigh\-dimensional continuous control using generalized advantage estimation\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1506.02438)Cited by:[§3\.2](https://arxiv.org/html/2608.26469#S3.SS2.p1.2)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.External Links:[Link](https://arxiv.org/abs/1707.06347)Cited by:[§4](https://arxiv.org/html/2608.26469#S4.p1.1)\.
- Silveret al\.\(2016\)D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. van den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot, S\. Dieleman, D\. Grewe, J\. Nham, N\. Kalchbrenner, I\. Sutskever, T\. Lillicrap, M\. Leach, K\. Kavukcuoglu, T\. Graepel, and D\. HassabisMastering the game of Go with deep neural networks and tree search\.Nature529\(7587\),pp\. 484–489\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/nature16961)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1)\.
- Sovianyet al\.\(2022\)P\. Soviany, R\. T\. Ionescu, P\. Rota, and N\. SebeCurriculum Learning: A Survey\.International Journal of Computer Vision130\(6\),pp\. 1526–1565\.External Links:ISSN 1573\-1405,[Document](https://dx.doi.org/10.1007/s11263-022-01611-x)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.Second edition,The MIT Press\.External Links:[Link](http://incompleteideas.net/book/the-book-2nd.html)Cited by:[§2\.1](https://arxiv.org/html/2608.26469#S2.SS1.SSS0.Px4.p1.1)\.
- Svetliket al\.\(2017\)M\. Svetlik, M\. Leonetti, J\. Sinapov, R\. Shah, N\. Walker, and P\. StoneAutomatic Curriculum Graph Generation for Reinforcement Learning Agents\.Proceedings of the AAAI Conference on Artificial Intelligence31\(1\)\.External Links:ISSN 2374\-3468, 2159\-5399,[Document](https://dx.doi.org/10.1609/aaai.v31i1.10933)Cited by:[§2\.2](https://arxiv.org/html/2608.26469#S2.SS2.p1.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Tanet al\.\(2023\)Z\. Tan, K\. Wang, and X\. WangC\-procgen: empowering procgen with controllable contexts\.External Links:2311\.07312,[Link](https://arxiv.org/abs/2311.07312)Cited by:[§C\.3\.2](https://arxiv.org/html/2608.26469#A3.SS3.SSS2.p2.1)\.
- Teohet al\.\(2024\)J\. Teoh, W\. Li, and P\. VarakanthamImproving environment novelty quantification for effective unsupervised environment design\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p2.1)\.
- Tiddet al\.\(2021\)B\. Tidd, N\. Hudson, and A\. CosgunGuided curriculum learning for walking over complex terrain\.External Links:2010\.03848,[Link](https://arxiv.org/abs/2010.03848)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p2.1)\.
- Tobinet al\.\(2017\)J\. Tobin, R\. Fong, A\. Ray, J\. Schneider, W\. Zaremba, and P\. AbbeelDomain randomization for transferring deep neural networks from simulation to the real world\.In2017 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 23–30\.External Links:[Document](https://dx.doi.org/10.1109/IROS.2017.8202133)Cited by:[§4](https://arxiv.org/html/2608.26469#S4.p2.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Tzannetoset al\.\(2024\)G\. Tzannetos, P\. Kamalaruban, and A\. SinglaProximal curriculum with task correlations for deep reinforcement learning\.InProceedings of the Thirty\-Third International Joint Conference on Artificial Intelligence,IJCAI ’24\.External Links:ISBN 978\-1\-956792\-04\-1,[Link](https://doi.org/10.24963/ijcai.2024/556),[Document](https://dx.doi.org/10.24963/ijcai.2024/556)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)L\. Wang, Z\. Xu, P\. Stone, and X\. XiaoGACL: grounded adaptive curriculum learning with active task and performance monitoring\.In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),Vol\.,pp\. 591–596\.External Links:[Document](https://dx.doi.org/10.1109/IROS60139.2025.11246910)Cited by:[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2019\)R\. Wang, J\. Lehman, J\. Clune, and K\. O\. StanleyPaired open\-ended trailblazer \(POET\): endlessly generating increasingly complex and diverse learning environments and their solutions\.arXiv preprint arXiv:1901\.01753\.Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p2.1),[§4\.2](https://arxiv.org/html/2608.26469#S4.SS2.p1.1),[§5](https://arxiv.org/html/2608.26469#S5.SS0.SSS0.Px2.p3.1)\.
- Wanget al\.\(2022\)X\. Wang, Y\. Chen, and W\. ZhuA survey on curriculum learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence44\(9\),pp\. 4555–4576\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2021.3069908)Cited by:[§1](https://arxiv.org/html/2608.26469#S1.p1.1)\.

## Appendix ADetailed Algorithm ofPATH

We provide the full pseudocode ofPATHbelow \(Algorithm 2\)\. For clarity, we define several notations used in the algorithm:Src⁡\(𝒢\)\\mathrm\{Src\}\(\\mathcal\{G\}\)denotes the set of source \(root\) nodes of the curriculum DAG𝒢\\mathcal\{G\}, i\.e\., nodes with no incoming edges;Sink⁡\(𝒢\)\\mathrm\{Sink\}\(\\mathcal\{G\}\)denotes the set of sink \(leaf\) nodes with no outgoing edges;Succ𝒢​\(θ\)\\mathrm\{Succ\}\_\{\\mathcal\{G\}\}\(\\theta\)denotes the set of immediate successors of nodeθ\\thetain𝒢\\mathcal\{G\}; andPLRScore⁡\(θ\)\\mathrm\{PLRScore\}\(\\theta\)computes the PLR positive value lossS⁡\(θ\)S\(\\theta\)as defined in Section[3\.2](https://arxiv.org/html/2608.26469#S3.SS2)\.

Algorithm 2PATH:Random→\\rightarrowActive1:Input:curriculum DAG

𝒞=\(Θ,ℰ\)\\mathcal\{C\}=\(\\Theta,\\mathcal\{E\}\)with source set

Src⁡\(𝒞\)\\mathrm\{Src\}\(\\mathcal\{C\}\), sink set

Sink⁡\(𝒞\)\\mathrm\{Sink\}\(\\mathcal\{C\}\), and successor set

Succ𝒞​\(θ\)\\mathrm\{Succ\}\_\{\\mathcal\{C\}\}\(\\theta\); buffer size

NN; minibatch size

BB; early stop reward

εes\\varepsilon\_\{\\mathrm\{es\}\}; sampling patience

εp\\varepsilon\_\{\\mathrm\{p\}\}\(Random only\); transition step

TswitchT\_\{\\mathrm\{switch\}\}; replay rate

pp\(Active only\); top\-

kkparameter

kk\(Active only\); Regret score

S​\(θ\)=PLRScore​\(θ,π\)S\(\\theta\)=\\textsc\{PLRScore\}\(\\theta,\\pi\)\(Active only\)\.

2:State:policy

π\\pi; stage

∈\{Random,Active\}\\in\\\{\{\\color\[rgb\]\{0,0,1\}\\textsc\{Random\}\},\{\\color\[rgb\]\{0,0\.6,0\}\\textsc\{Active\}\}\\\}; buffer entries

\{\(bi,wi\)\}i=1N\\\{\(b\_\{i\},w\_\{i\}\)\\\}\_\{i=1\}^\{N\}with current level

bi∈Θb\_\{i\}\\in\\Thetaand weight

wi\>0w\_\{i\}\>0; counters

\{ci\}i=1N\\\{c\_\{i\}\\\}\_\{i=1\}^\{N\}\(Random only\)\.

3:Init:stage

←Random\\leftarrow\{\\color\[rgb\]\{0,0,1\}\\textsc\{Random\}\}; for

i∈\[N\]i\\in\[N\]:

bi∼Unif⁡\(Src⁡\(𝒞\)\)b\_\{i\}\\sim\\mathrm\{Unif\}\(\\mathrm\{Src\}\(\\mathcal\{C\}\)\),

wi←1w\_\{i\}\\leftarrow 1,

ci←0c\_\{i\}\\leftarrow 0,

n←0n\\leftarrow 0,

s←0s\\leftarrow 0\.

4:whilenot convergeddo

5:ifstage

=Random=\{\\color\[rgb\]\{0,0,1\}\\textsc\{Random\}\}then

6:Sample indicesI∼Unif​\(\[N\]\)BI\\sim\\mathrm\{Unif\}\(\[N\]\)^\{B\}\.

7:foreachi∈Ii\\in Ido

8:

θ←bi\\theta\\leftarrow b\_\{i\}; rollout onθ\\theta; updateπ\\pi; compute average returnJ^​\(θ\)\\widehat\{J\}\(\\theta\);ci←ci\+1c\_\{i\}\\leftarrow c\_\{i\}\+1\.

9:if

J^​\(θ\)≥εes\\widehat\{J\}\(\\theta\)\\geq\\varepsilon\_\{\\mathrm\{es\}\}orci≥εpc\_\{i\}\\geq\\varepsilon\_\{\\mathrm\{p\}\}then

10:

bi←\{Unif⁡\(Src⁡\(𝒞\)\)θ∈Sink⁡\(𝒞\)​or​ci≥εpUnif​\(Succ𝒞​\(θ\)\)otherwiseb\_\{i\}\\leftarrow\\begin\{cases\}\\mathrm\{Unif\}\(\\mathrm\{Src\}\(\\mathcal\{C\}\)\)&\\theta\\in\\mathrm\{Sink\}\(\\mathcal\{C\}\)\\ \\text\{or\}\\ c\_\{i\}\\geq\\varepsilon\_\{\\mathrm\{p\}\}\\\\ \\mathrm\{Unif\}\(\\mathrm\{Succ\}\_\{\\mathcal\{C\}\}\(\\theta\)\)&\\text\{otherwise\}\\end\{cases\}
11:

n←n\+𝕀\{θ∈Sink\(𝒞\)orci≥εp\}n\\leftarrow n\+\\mathbb\{I\}\\\{\\theta\\in\\mathrm\{Sink\}\(\\mathcal\{C\}\)\\ \\text\{or\}\\ c\_\{i\}\\geq\\varepsilon\_\{\\mathrm\{p\}\}\\\};ci←0c\_\{i\}\\leftarrow 0;wi←1w\_\{i\}\\leftarrow 1\.

12:

s←s\+1s\\leftarrow s\+1\.

13:if

smod1000=0​ands\\bmod 1000=0\\ \\mathrm\{and\}n≥Tswitchn\\geq T\_\{\\mathrm\{switch\}\}then

14:stage

←Active\\leftarrow\{\\color\[rgb\]\{0,0\.6,0\}\\textsc\{Active\}\};setwi←1w\_\{i\}\\leftarrow 1for alliiand discard counters\{ci\}\\\{c\_\{i\}\\\}\.

15:ifstage

=Active=\{\\color\[rgb\]\{0,0\.6,0\}\\textsc\{Active\}\}then

16:ifwith probabilityppthen

17:Updatep⁡\(i\)=wi/∑j=1Nwjp\(i\)=w\_\{i\}/\\sum\_\{j=1\}^\{N\}w\_\{j\}; sampleI∼pBI\\sim p^\{B\}; setE←∅E\\leftarrow\\emptyset\.

18:foreachi∈Ii\\in Ido

19:

θ←bi\\theta\\leftarrow b\_\{i\}; rollout onθ\\theta; updateπ\\pi; compute average returnJ^​\(θ\)\\widehat\{J\}\(\\theta\); setwi←S⁡\(θ\)w\_\{i\}\\leftarrow S\(\\theta\)\.

20:if

J^​\(θ\)≥εes\\widehat\{J\}\(\\theta\)\\geq\\varepsilon\_\{\\mathrm\{es\}\}then

21:

E←E∪\{\(i,wi\)\}E\\leftarrow E\\cup\\\{\(i,w\_\{i\}\)\\\}\.

22:

m←min⁡\{\|E\|,k\}m\\leftarrow\\min\\\{\|E\|,k\\\};Itop←I\_\{\\mathrm\{top\}\}\\leftarrowthemmindices inEEwith largest weights; pad toLLof sizeBB\.

23:foreachi∈Li\\in Ldo

24:

θ←bi\\theta\\leftarrow b\_\{i\}; sampleθ\+←\{Unif⁡\(Src⁡\(𝒞\)\)θ∈Sink⁡\(𝒞\)Unif​\(Succ𝒞​\(θ\)\)otherwise\\theta^\{\+\}\\leftarrow\\begin\{cases\}\\mathrm\{Unif\}\(\\mathrm\{Src\}\(\\mathcal\{C\}\)\)&\\theta\\in\\mathrm\{Sink\}\(\\mathcal\{C\}\)\\\\ \\mathrm\{Unif\}\(\\mathrm\{Succ\}\_\{\\mathcal\{C\}\}\(\\theta\)\)&\\text\{otherwise\}\\end\{cases\}; append\(θ\+,1\)\(\\theta^\{\+\},1\)in the buffer\.

25:else

26:Sampleθ1,…,θB∼Unif⁡\(Src⁡\(𝒞\)\)\\theta\_\{1\},\\dots,\\theta\_\{B\}\\sim\\mathrm\{Unif\}\(\\mathrm\{Src\}\(\\mathcal\{C\}\)\); append\(θt,1\)\(\\theta\_\{t\},1\)for allt∈\[B\]t\\in\[B\]\.

27:While buffer size\>N\>N, delete a minimum\-weight entry\.

## Appendix BImplementation Details

This section details \(i\) the environment configurations used in our experiments, \(ii\) the curriculum design and environment\-generation procedure instantiated by each baseline \(DR, PLR, ACCEL, andPATH\), and \(iii\) training information and hyperparameters used for all methods\. ForPATH, we explicitly construct a curriculum DAG that matches the edit\-based curriculum structure used implicitly by ACCEL; this alignment ensures comparisons isolate algorithmic differences in*how*the curriculum is exploited, rather than differences in the underlying curriculum itself\.

### B\.1MiniGrid

##### Environment

We consider a partially observable navigation task\([Dennis et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib9)\)on a15×1515\\times 15grid, implemented by[Chevalier\-Boisvert et al\. \(2018\)](https://arxiv.org/html/2608.26469#bib.bib23)\. Each cell is either a clutter \(block\), free space, the agent, or the goal\. An episode terminates when the agent reaches the goal or when the time limitTmax=250T\_\{\\max\}=250is reached\. If the agent reaches the goal at time stepTT, it receives a terminal reward1−T/Tmax1\-T/T\_\{\\max\}; otherwise the reward is00\. Observations are partial: the agent observes only a5×55\\times 5egocentric window in front of it, and actions move the agent by one cell in the four cardinal directions at each time step\. We consider an environment grid solved when the policy’s return exceeds 0\.

##### Curriculum

We compare against DR, PLR, and ACCEL under a common layout space and clutters budget\. LetΘ\\Thetadenote the set of all valid15×1515\\times 15grid layouts with at most6060interior clutters \(excluding boundary walls\), with a*fixed*agent start cell and a*fixed*goal cell chosen once at initialization and held constant thereafter\. DR and PLR treatΘ\\Thetaas an unstructured set and sample layouts i\.i\.d\.: they first sample a clutter countw∼Unif​\{0,…,60\}w\\sim\\mathrm\{Unif\}\\\{0,\\dots,60\\\}and then placewwclutters uniformly at random among interior cells, rejecting and resampling any placement that occupies the fixed start or goal\. In contrast, ACCEL induces an implicit curriculum through*edit\-based*layout evolution from an empty room\. To make this curriculum explicit \(and shared for fair comparison\), we instantiate a curriculum DAG𝒢=\(Θ,ℰ\)\\mathcal\{G\}=\(\\Theta,\\mathcal\{E\}\)whose source nodes is the empty room with the random start/goal, and whose edges encode minimal topologically continuous changes: forθ,θ′∈Θ\\theta,\\theta^\{\\prime\}\\in\\Theta, we include a directed edge\(θ→θ′\)∈ℰ\(\\theta\\\!\\to\\\!\\theta^\{\\prime\}\)\\in\\mathcal\{E\}if and only ifθ′\\theta^\{\\prime\}is obtained fromθ\\thetaby a single valid local edit, namely inserting one interior clutter while preserving validity with the fixed start/goal\. Denoting the out\-neighborhood by𝒩\+​\(θ\)=\{θ′∈Θ:\(θ→θ′\)∈ℰ\}\\mathcal\{N\}^\{\+\}\(\\theta\)=\\\{\\theta^\{\\prime\}\\in\\Theta:\(\\theta\\\!\\to\\\!\\theta^\{\\prime\}\)\\in\\mathcal\{E\}\\\}, exploration over this curriculum is a Markov process on𝒢\\mathcal\{G\}: starting at the source \(empty room\), the next layout is drawn uniformly from the outgoing edges, i\.e\.,P\(θ′∣θ\)=𝟏\{θ′∈𝒩\+\(θ\)\}/\|𝒩\+\(θ\)\|P\(\\theta^\{\\prime\}\\mid\\theta\)=\\mathbf\{1\}\\\{\\theta^\{\\prime\}\\in\\mathcal\{N\}^\{\+\}\(\\theta\)\\\}/\|\\mathcal\{N\}^\{\+\}\(\\theta\)\|\. Following ACCEL, we allow small stochastic edit distances by samplingK∼Unif​\{1,…,5\}K\\sim\\mathrm\{Unif\}\\\{1,\\dots,5\\\}and applyingKKsuccessive one\-edit transitions, equivalentlyθt\+1∼P\(K\)\(⋅∣θt\)\\theta\_\{t\+1\}\\sim P^\{\(K\)\}\(\\cdot\\mid\\theta\_\{t\}\)\. If𝒩\+​\(θt\)\\mathcal\{N\}^\{\+\}\(\\theta\_\{t\}\)is empty or the clutter number is up to the limit, the process restarts at the source node, ensuring sampled environments evolve via small, topologically continuous edits within the same layout space used by all baselines\.

### B\.2BipedalWalker

##### Environment

We use a modified version by[Parker\-Holder et al\. \(2022a\)](https://arxiv.org/html/2608.26469#bib.bib11)that enables active parameterization of the terrain complexity of BipedalWalker environment in OpenAI Gym\([Brockman et al\., 2016](https://arxiv.org/html/2608.26469#bib.bib21)\)\)\. The agent observes a 24\-dimensional proprioceptive state \(lidar readings, joint angles, and contact indicators\) without access to absolute position\. Actions are four continuous motor torques\. The agent receives a positive reward each step for forward progress and a penalty of−100\-100upon falling\. Successfully traversing the terrain within the 2000 step horizon typically yields a total return exceeding 300\. We consider a terrain instancesolvedif the policy achieves a return of at least 230\.

Table 3:Environment parameter ranges and editing details of ACCEL on BipedalWalker
##### Curriculum

We follow the ACCEL configuration for BipedalWalker\. Let𝒱⊂ℝd\\mathcal\{V\}\\subset\\mathbb\{R\}^\{d\}\(d=8d=8\) denote the set of all valid terrain\-parameter vectorsv∈ℝdv\\in\\mathbb\{R\}^\{d\}under the coordinate\-wise bounds0≤vi≤vimax0\\leq v\_\{i\}\\leq v\_\{i\}^\{\\max\}\(See MAX VALUE in Table[3](https://arxiv.org/html/2608.26469#A2.T3)\), and letssbe a random seed\. DR and PLR treat𝒱\\mathcal\{V\}as an unstructured design space and sample environment instances i\.i\.d\. by drawingvvandssdirectly\. In contrast, ACCEL induces an implicit curriculum through*edit\-based*evolution ofvvstarting fromv\(0\)v^\{\(0\)\}from the initialized parameter\. To make this curriculum explicit, we instantiate a curriculum DAG𝒢=\(Θ,ℰ\)\\mathcal\{G\}=\(\\Theta,\\mathcal\{E\}\)whose nodes are environment instancesθ=\(v,s\)∈Θ:=𝒱×𝒮\\theta=\(v,s\)\\in\\Theta:=\\mathcal\{V\}\\times\\mathcal\{S\}, and whose edges encode minimal ACCEL mutations: forθ=\(v,s\)\\theta=\(v,s\)andθ′=\(v′,s′\)\\theta^\{\\prime\}=\(v^\{\\prime\},s^\{\\prime\}\), we include a directed edge\(θ→θ′\)∈ℰ\(\\theta\\\!\\to\\\!\\theta^\{\\prime\}\)\\in\\mathcal\{E\}if and only ifθ′\\theta^\{\\prime\}is obtained fromθ\\thetaby a single valid coordinate edit, i\.e\., selecting an indexj∈\{1,…,d\}j\\in\\\{1,\\dots,d\\\}and setting

vj′=min\{vj\+Δj,\(vmax\)j\},vi′=vi\(i≠j\),v^\{\\prime\}\_\{j\}=\\min\\\{v\_\{j\}\+\\Delta\_\{j\},\\ \(v\_\{\\max\}\)\_\{j\}\\\},\\qquad v^\{\\prime\}\_\{i\}=v\_\{i\}\\ \\ \(i\\neq j\),whereΔj\\Delta\_\{j\}is the fixed edit size for coordinatejj\(Table 9\)\. Denoting the out\-neighborhood by𝒩\+​\(θ\)=\{θ′∈Θ:\(θ→θ′\)∈ℰ\}\\mathcal\{N\}^\{\+\}\(\\theta\)=\\\{\\theta^\{\\prime\}\\in\\Theta:\(\\theta\\\!\\to\\\!\\theta^\{\\prime\}\)\\in\\mathcal\{E\}\\\}, exploration over this curriculum is a Markov process on𝒢\\mathcal\{G\}: given the current nodeθt\\theta\_\{t\}, the next node is drawn uniformly from outgoing edges, i\.e\.,P\(θ′∣θ\)=𝟏\{θ′∈𝒩\+\(θ\)\}/\|𝒩\+\(θ\)\|P\(\\theta^\{\\prime\}\\mid\\theta\)=\\mathbf\{1\}\\\{\\theta^\{\\prime\}\\in\\mathcal\{N\}^\{\+\}\(\\theta\)\\\}/\|\\mathcal\{N\}^\{\+\}\(\\theta\)\|with one random seed\. Following ACCEL, we allow small stochastic edit distances by samplingK∼Unif​\{1,2,3\}K\\sim\\mathrm\{Unif\}\\\{1,2,3\\\}and applyingKKsuccessive one\-edit transitions, equivalentlyθt\+1∼P\(K\)\(⋅∣θt\)\\theta\_\{t\+1\}\\sim P^\{\(K\)\}\(\\cdot\\mid\\theta\_\{t\}\)\. Small edit sizes yield functional, local curriculum progress, while the fixed mutation scale introduces controlled variation that promotes generalization across nearby configurations\.

### B\.3Training Information

For MiniGrid, each run \(one seed per method\) uses a single NVIDIA Tesla V100 GPU\. Training for up to2020k PPO updates takes approximately5050hours per seed for DR andPATH:Random, and approximately100100hours per seed for PLR and ACCEL\.

For BipedalWalker, each run uses a single NVIDIA A40 GPU\. Training for up to2020k PPO updates takes approximately6060hours per seed for DR andPATH:Random,100100hours per seed forPATH\(transitioning fromPATH:RandomtoPATH:Activeat88k updates\), and approximately160160hours per seed for PLR, ACCEL,PATH:Active\.

### B\.4Hyperparameters

We inherit identical hyperparameters for PPO training and baselines \(DR,PLR,ACCEL\) configurations from\([Parker\-Holder et al\., 2022a](https://arxiv.org/html/2608.26469#bib.bib11)\)\. We conclude them withPATH’s hyperparameters in Table[9](https://arxiv.org/html/2608.26469#A3.T9)\.

## Appendix CFull Experiments Details

In this section, we detail the construction of the MiniGrid and BipedalWalker test datasets and report the corresponding results referenced in Section[4](https://arxiv.org/html/2608.26469#S4)\.

### C\.1MiniGrid

#### C\.1\.1Robustness Test

We use the fixed, challenging MiniGrid evaluation suite introduced by[Jiang et al\. \(2021a\)](https://arxiv.org/html/2608.26469#bib.bib17), shown in Figure[12](https://arxiv.org/html/2608.26469#A3.F12)\. For each layout, the agent’s start cell, goal cell, and initial orientation are held fixed across all evaluation episodes\. For each method, we evaluate thenondeterministicpolicy2020k PPO updates over 100 episodes and report the corresponding solved rate\. Detailed results are provided in Table[4](https://arxiv.org/html/2608.26469#A3.T4)\. Overall,PATHis both stable and consistently outperforms the baselines, achieving perfect solved rates on most test grids\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/robust_envs_grid.png)Figure 12:Specific layouts for each MiniGrid environment in the robustness test set\.Table 4:Solved rates after2020k PPO updates \(mean±\\pmstandard error\) over55independent runs for each method on each fixed MiniGrid layout \(robustness test\)\. For each run, we evaluate the final policy for100100episodes per layout\. Bold entries are within one standard error of the best mean\.
#### C\.1\.2Generalization Test

To assess policy generalization, we construct a held\-out test set by uniformly sampling10001000valid grid layouts from the environment spaceΘ\\Theta\. Figure[13](https://arxiv.org/html/2608.26469#A3.F13)shows that these test grids are broadly distributed across the space under the two complexity dimensions\. For evaluation, each method’s trained policy is executed deterministically on every test grid for a single episode, and we report the overall solve rate across the full test set\. The results are summarized in Table[1](https://arxiv.org/html/2608.26469#S4.T1):PATHsolves nearly all test environments, indicating strong generalization\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/mg_general_test_distribution.png)Figure 13:Distribution of 1000 valid grids uniformly sampled fromΘ\\Theta\(clutter vs\. shortest\-path length\)\.

### C\.2Bipedalwalker

#### C\.2\.1Robustness Test

To evaluate robustness on BipedalWalker, we construct a test suite comprising six environment classes \(examples in Figure[14](https://arxiv.org/html/2608.26469#A3.F14)\)\.BasicandHardCoreare the standard OpenAI Gym environments\([Brockman et al\., 2016](https://arxiv.org/html/2608.26469#bib.bib21)\)\. The remaining four classes follow the fixed ACCEL configurations of[Parker\-Holder et al\. \(2022a\)](https://arxiv.org/html/2608.26469#bib.bib11):Stairsuses stair height\[2,2\]\[2,2\]with55steps;PitGapfixes the pit gap to\[5,5\]\[5,5\];Stumpfixes stump height to\[2,2\]\[2,2\]; andRoughnesssets ground roughness to55\. For each class, we generate128128terrains using distinct seeds and evaluate the policydeterministicallyfor one episode per terrain to obtain returns\. Detailed results are reported in Table[5](https://arxiv.org/html/2608.26469#A3.T5)\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_robust_examples.png)Figure 14:Examples of BipedalWalker terrain classes used for robustness testing\.Table 5:Average returns after2020k PPO updates \(mean±\\pmstandard error\) over55independent runs for each method on each robustness test class of BipedalWalker\. For each run, we evaluate the final policy for11episodes for128128terrains of each class\. Bold entries are within one standard error of the best mean\.
#### C\.2\.2Generalization Test

We sample10001000terrains from the BipedalWalker environment spaceΘ\\Thetaby drawing terrain\-parameter vectors at random and pairing each draw with a random seed, forming a held\-out generalization test set \(examples in Figure[15](https://arxiv.org/html/2608.26469#A3.F15)\)\. For this dataset, we sample parameters directly from their continuous ranges \(without one input of intervals\), so each terrain is specified by an exact55\-dimensional parameter vector\. We evaluate each method by running the learned policy*deterministically*for one episode on every test terrain, and report aggregate statistics in Table[2](https://arxiv.org/html/2608.26469#S4.T2)\. Overall, this test set contains many highly challenging, often unsolved terrains, whereasPATHconsistently pushes the learning frontier, achieving higher mean return and solving a larger fraction of terrains than the baselines\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/bw_general_envs_example.png)Figure 15:Examples from the BipedalWalker generalization test set of 1000 randomly sampled terrains\.

### C\.3Leaper

#### C\.3\.1Environment Description

Leaper[16](https://arxiv.org/html/2608.26469#A3.F16)is a Procgen game\([Cobbe et al\., 2020](https://arxiv.org/html/2608.26469#bib.bib33)\)in which a frog agent must cross a road \(dodging cars\) and a river \(hopping on logs\) to reach a goal lane\. The state space is represented by64×64×364\\times 64\\times 3pixel observations, and the action space follows the 15\-dimensional Procgen game setting\. It is a sparse\-reward problem: the agent receives a reward of 0 or 10 upon success\. Unlike MiniGrid or BipedalWalker, Leaper does not expose an explicit editing structure or difficulty parameter\. Instead, difficulty emerges naturally from four observable environmental axes: \(1\) maximum car speed, \(2\) maximum log speed, \(3\) number of road lanes, and \(4\) number of water lanes\.

This benchmark directly addresses the implicit parameterization setting described in Section[1](https://arxiv.org/html/2608.26469#S1)\(iii\)—one that is more representative of realistic scenarios where environment parameters are not explicitly accessible\.

#### C\.3\.2Applying PATH

We assume that increasing each axis leads to a harder environment, providing a natural difficulty ordering\. We treat each axis as a curriculum dimension and assume monotone difficulty along it\. This is howPATHis applied in the implicit case: by identifying observable structure and using it as a reasonable prior, without requiring a strictly guaranteed monotone DAG\.

We represent the curriculum DAG using difficulty buckets, where each bucket is a 4\-dimensional tuple\. We randomly sample 300 Leaper environments as the training set\. To construct buckets, we leverage CProcgen\([Tan et al\., 2023](https://arxiv.org/html/2608.26469#bib.bib53)\), a C\+\+ Procgen backend that exposes the context parameters of each Procgen environment, allowing us to directly read out the four difficulty axes\. We emphasize that this step can in principle be performed without explicit parameterization—observable environment statistics suffice\. We compute statistics for each dimension, then discretize each axis into 3 levels \(easy/medium/hard\) with an approximately equal number of environments per level, yielding34=813^\{4\}=81buckets\. For instance, the number of car lanes ranges from 1 to 3 across the three difficulty levels along that dimension\. The induced DAG starts at bucket\[0,0,0,0\]\[0,0,0,0\]; oncePATH’s early\-stop criterion is satisfied, it randomly selects a neighboring bucket with one dimension incremented and samples uniformly from the environments within it\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/leaper.jpg)Figure 16:An example of the Leaper environment from Procgen\.
#### C\.3\.3Experiment Configuration and results

We use 300 training environments across 81 buckets, 32 parallel workers for PPO, 1000 total PPO update steps, rollout length 256, replay buffer size 30, early\-stop threshold 10\.0, patience 20 updates, and a phase shift after 500 PPO updates forPATH\. We adopt the PPO hyperparameter settings from the PLR paper\. We comparePATH, ACCEL, PLR, and DR across 5 seeds\.

Table 6:Leaper generalization results\. Evaluated on 100 held\-out test environments, averaged over 5 seeds\.PATHachieves the highest average solved rate with the lowest standard error across all methods\. One notable observation is that DR is a strong baseline in this setting, likely due to the relatively small training set \(300 environments\), where uniform sampling already provides substantial coverage\. However,PATHstill outperforms DR, demonstrating that even when the curriculum DAG is not explicitly given but is instead constructed from observable difficulty axes,PATH’s structured traversal yields meaningful gains over unstructured and replay\-only methods\.

### C\.4Ablations

We conduct several ablation experiments to provide a more comprehensive study ofPATH\.

#### C\.4\.1Component ablation:PATH:Randomvs\.PATH:Active\.

Our first ablation disentangles the contributions ofPATH:RandomandPATH:Active\. We report training curves on the robustness suite in Figure[17](https://arxiv.org/html/2608.26469#A3.F17), and generalization performance in Figure[11](https://arxiv.org/html/2608.26469#S4.F11)\. The analysis in Section[4\.3](https://arxiv.org/html/2608.26469#S4.SS3)show thatPATHbenefits from*both*phases:PATH:Randomimproves broad curriculum coverage, whilePATH:Activefurther improves sample efficiency by concentrating training on environments with remaining learning potential\.

![Refer to caption](https://arxiv.org/html/2608.26469v1/figures/ablation_training_curves.png)Figure 17:Performance on BipedalWalker robustness test environments during training \(mean±\\pmstandard error\), evaluated every11k PPO updates\. We compare threePATHvariants:PATHrunsPATH:RANDOMuntil88k updates and then switches toPATH:ACTIVE;PATH:RANDOMkeeps the random phase throughout training;PATH:ACTIVEuses the active phase from the beginning\. The vertical dashed line marks the88k transition point forPATH\.
#### C\.4\.2Hyperparameter ablation: early stopping and sampling patience\.

Our second ablation studies sensitivity to two hyperparameters: the early stop reward thresholdℰes\\mathcal\{E\}\_\{\\mathrm\{es\}\}and the sampling patienceℰp\\mathcal\{E\}\_\{p\}\. We denote aPATHinstantiation byPATHℰesℰp\.\\textsc\{PATH\}^\{\\mathcal\{E\}\_\{p\}\}\_\{\\mathcal\{E\}\_\{\\mathrm\{es\}\}\}\.The default configuration used in this ablation comparisons corresponds toPATH15020\\textsc\{PATH\}^\{20\}\_\{150\}\. We compare it againstPATH15030\\textsc\{PATH\}^\{30\}\_\{150\}\(larger patience\),PATH10020\\textsc\{PATH\}^\{20\}\_\{100\}andPATH20020\\textsc\{PATH\}^\{20\}\_\{200\}\(weaker/stronger early stop thresholds\), andPATHNone20\\textsc\{PATH\}^\{20\}\_\{\\mathrm\{None\}\}\(no early stopping\)\. InPATHNone20\\textsc\{PATH\}^\{20\}\_\{\\mathrm\{None\}\}, each curriculum node is sampled exactlyℰp=20\\mathcal\{E\}\_\{p\}=20times before we proceed, regardless of observed returns\.

For fair comparison, all methods use the samePATH:Randomphase for the first88k PPO updates, then transfer toPATH:Activeand continue training until2020k PPO updates\. We report robustness results in Table[7](https://arxiv.org/html/2608.26469#A3.T7)and generalization results in Table[8](https://arxiv.org/html/2608.26469#A3.T8)\. The main observation is thatPATHNone20\\textsc\{PATH\}^\{20\}\_\{\\mathrm\{None\}\}performs substantially worse on both robustness and generalization\. This is consistent with the role of early stopping inPATH: without it, a large fraction of the training budget is spent repeatedly revisiting nodes that are already effectively solved, which reduces coverage of the curriculum graph and slows progress toward unmastered regions\.

In contrast, other variants—PATH10020\\textsc\{PATH\}^\{20\}\_\{100\},PATH15020\\textsc\{PATH\}^\{20\}\_\{150\},PATH20020\\textsc\{PATH\}^\{20\}\_\{200\}, andPATH15030\\textsc\{PATH\}^\{30\}\_\{150\}—achieve broadly similar performance, with the defaultPATH15020\\textsc\{PATH\}^\{20\}\_\{150\}attaining the best overall aggregate score in our study\. This indicates an inherent trade\-off in selectingℰes\\mathcal\{E\}\_\{\\mathrm\{es\}\}, as discussed in Section[3\.1](https://arxiv.org/html/2608.26469#S3.SS1)\. Increasing patience fromℰp=20\\mathcal\{E\}\_\{p\}=20toℰp=30\\mathcal\{E\}\_\{p\}=30\(i\.e\.,PATH15030\\textsc\{PATH\}^\{30\}\_\{150\}\) is slightly worse than the default, supporting our design choice thatℰp=20\\mathcal\{E\}\_\{p\}=20is an effective upper bound for maintaining smooth curriculum progression without overspending on mastered nodes\.

Overall, aside from the no early stop setting,PATHis relatively insensitive to reasonable choices ofℰes\\mathcal\{E\}\_\{\\mathrm\{es\}\}andℰp\\mathcal\{E\}\_\{p\}, and these configurations remain competitive and consistently stronger than baselines such as ACCEL\. This ablation therefore supportsPATHas a stable and effective curriculum learning method\.

Table 7:Average returns after2020k PPO updates \(mean±\\pmstandard error\) across55independent runs for each ablation variant on the BipedalWalker robustness test classes\. For each run, we evaluate the final policy for one episode on each of128128terrains per class\. Bold entries indicate results within one standard error of the best mean\.PATH15020\\textsc\{PATH\}^\{20\}\_\{150\}is the default algorithm\.Table 8:Generalization test returns and solved rates at2020k updates \(mean±\\pmstandard error\) for55runs of each ablation variant on BipedalWalker\. Bold values are within one standard error of the best mean\.PATH15020\\textsc\{PATH\}^\{20\}\_\{150\}is the default algorithm\.Table 9:Hyperparameters for RL training and curriculum algorithms\.
#### C\.4\.3Robustness to DAG Misspecification

To directly evaluatePATH’s robustness to noisy or incorrect curriculum edges, we introduce a*perturbation probability*pperturbp\_\{\\text\{perturb\}\}: at each successor selection after the early\-stop criterion is met, with probabilitypperturbp\_\{\\text\{perturb\}\}a uniformly random node from the full graph space is selected instead of the curriculum successor, explicitly breaking DAG structure\. All results are averaged over 5 seeds\.PATHshifts phase at 5k steps; total training is 10k updates\.

Table 10:MiniGrid — Robustness Solved Rate at 10k \(same setting as Table[4](https://arxiv.org/html/2608.26469#A3.T4)\)\.Table 11:BipedalWalker — Mean Robustness Return at 10k \(same setting as Table[5](https://arxiv.org/html/2608.26469#A3.T5)\)\.At 30% perturbation,PATHremains comparable to ACCEL on MiniGrid \(0\.530\.53vs\.0\.560\.56, within one standard error\) and exceeds ACCEL on BipedalWalker \(168\.24168\.24vs\.142\.31142\.31\)\. At 50% perturbation—half of all curriculum edges replaced with random jumps—PATHstill substantially outperforms DR and PLR on both benchmarks\. Performance degrades gradually rather than catastrophically, consistent with the two\-level safeguard:εes\\varepsilon\_\{\\text\{es\}\}advances cheaply when a successor is unexpectedly easy, andεp\\varepsilon\_\{p\}bounds budget waste when a successor provides no transfer benefit, reinitializing the pointer after at most 20–25 steps \(Table[9](https://arxiv.org/html/2608.26469#A3.T9)\)\. In the adversarial limit where every edge is uninformative,PATHdegrades toPATH:Random, which already substantially outperforms all baselines\.

#### C\.4\.4Two\-Phase Validation on MiniGrid

In the main experiments \(Section[4](https://arxiv.org/html/2608.26469#S4)\),PATHnever switches fromPATH:RandomtoPATH:Activewithin 20k updates on MiniGrid because near\-ceiling performance \(99\.4% generalization solved rate\) is achieved entirely during the random phase\. To directly validate that the two\-stage design provides value on MiniGrid when the random phase does not suffice, we run a shortened experiment: 10k total updates with phase shift at 5k, so that both phases must operate within the training budget\.

Table 12:MiniGrid — Robustness Solved Rate under shortened horizon \(5 seeds\)\.At 5k updates \(still in the random phase\),PATHmatchesPATH:Randomexactly and already outperforms ACCEL andPATH:Active\. By 10k updates,PATH\(0\.76\) clearly outperforms bothPATH:Random\(0\.70\) andPATH:Active\(0\.59\), confirming thatPATH:Activeadds measurable value once the random phase saturates\. The original 20k experiment simply did not require switching because near\-ceiling performance was already achieved\.

#### C\.4\.5BipedalWalker Generalization in the Solvable Subspace

The full BipedalWalker parameter space\[10,10,5,5,9\]\[10,10,5,5,9\]contains a significant portion of inherently unsolvable configurations\. To better isolate meaningful differences, we restrict the upper parameter limits to\[6,6,3,3,6\]\[6,6,3,3,6\]—a more tractable subspace—and evaluate on 1000 uniformly sampled environments at 20k updates, averaged over 5 seeds\.

Table 13:BipedalWalker generalization in the cropped solvable subspace\.PATHachieves a 97% improvement in solved rate and 66% improvement in mean score over ACCEL, confirming that the relative gains observed in the full parameter space are large and significant when restricted to the solvable subspace\.