Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL

arXiv cs.LG Papers
reinforcement-learning offline-rl goal-conditioned long-horizon value-learning divide-and-conquer binary-tree

Summary

The paper introduces DCRL, a divide-and-conquer approach for offline goal-conditioned reinforcement learning that reduces error accumulation in long-horizon tasks via recursive binary tree decomposition, achieving improved performance on OGBench benchmarks.

arXiv:2609.02237v1 Announce Type: new Abstract: Scaling offline goal-conditioned reinforcement learning (GCRL) to long-horizon tasks is difficult because (1) long-range value learning depends on shorter-range estimates that may still be inaccurate, and (2) max-based value backups can amplify overestimation through repeated propagation. We propose DCRL (Divide-and-Conquer RL), which recursively decomposes each trajectory segment into a balanced binary tree and trains the values from leaves to root. Each parent is therefore updated only after its children, using an exact factorization of the observed route rather than selecting among noisy alternatives. Since this objective learns values along demonstrated routes that are not necessarily optimal, DCRL jointly propagates values across trajectories to discover shorter routes. Thanks to the balanced binary tree, DCRL reduces worst-case bootstrap depth from linear to logarithmic, and this shorter dependency structure empirically corresponds to much slower error accumulation. Across diverse goal-reaching tasks, DCRL substantially outperforms prior flat offline GCRL methods, and on the five most challenging long-horizon OGBench tasks, it improves the best prior average score from 55 to 64, surpassing all flat and hierarchical baselines.
Original Article
View Cached Full Text

Cached at: 09/03/26, 06:17 AM 

# Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
Source: [https://arxiv.org/abs/2609.02237](https://arxiv.org/abs/2609.02237)
[View PDF](https://arxiv.org/pdf/2609.02237)

> Abstract:Scaling offline goal\-conditioned reinforcement learning \(GCRL\) to long\-horizon tasks is difficult because \(1\) long\-range value learning depends on shorter\-range estimates that may still be inaccurate, and \(2\) max\-based value backups can amplify overestimation through repeated propagation\. We propose DCRL \(Divide\-and\-Conquer RL\), which recursively decomposes each trajectory segment into a balanced binary tree and trains the values from leaves to root\. Each parent is therefore updated only after its children, using an exact factorization of the observed route rather than selecting among noisy alternatives\. Since this objective learns values along demonstrated routes that are not necessarily optimal, DCRL jointly propagates values across trajectories to discover shorter routes\. Thanks to the balanced binary tree, DCRL reduces worst\-case bootstrap depth from linear to logarithmic, and this shorter dependency structure empirically corresponds to much slower error accumulation\. Across diverse goal\-reaching tasks, DCRL substantially outperforms prior flat offline GCRL methods, and on the five most challenging long\-horizon OGBench tasks, it improves the best prior average score from 55 to 64, surpassing all flat and hierarchical baselines\.

## Submission history

From: Hyeonseong Jeon \[[view email](https://arxiv.org/show-email/22ba45ee/2609.02237)\] **\[v1\]**Wed, 2 Sep 2026 07:45:16 UTC \(6,257 KB\)

Similar Articles