PROWL: Prioritized Regret-Driven Optimization for World Model Learning

arXiv cs.LG Papers
world-models adversarial-training reinforcement-learning video-generation curriculum-learning robust-optimization diffusion-models

Summary

Introduces PROWL, a prioritized regret-driven optimization framework that uses an adversarial curriculum to improve diffusion-based world model robustness by focusing on high-error trajectories, achieving better performance on out-of-distribution scenarios in MineRL.

arXiv:2605.18803v1 Announce Type: new Abstract: Modern action-conditioned video world models achieve strong short-horizon visual realism, yet remain unreliable on rare, interaction-critical transitions that dominate downstream planning and policy performance. Because passive demonstration data systematically under-samples these high-impact regimes, improving robustness requires actively eliciting model failures rather than relying on their natural occurrence. We introduce a KL-constrained adversarial curriculum in which a policy is trained to expose high-error trajectories of a diffusion-based world model while remaining close to the behavior distribution. The world model is continuously fine-tuned on these adversarially discovered trajectories, yielding an adversarial training loop that converts rare failures into a stable, near-distribution training signal without drifting into out-of-distribution exploitation. To maintain pressure on unresolved weaknesses as the model improves, we propose a Prioritized Adversarial Trajectory (PAT) buffer that re-ranks trajectories based on prediction error, action fidelity, and learning progress, focusing training on unresolved failure modes rather than repeatedly revisiting solved cases. We implement our approach in the MineRL framework and evaluate it on held-out out-of-distribution trajectories; PROWL improves robustness over models trained on passive data alone, reveals reward-hacking behaviors under weak behavioral constraints, and demonstrates that effective adversarial world-model training critically depends on balancing exploratory failure discovery with explicit behavioral regularization. Our results suggest that scalable world models benefit not only from larger datasets, but also from selectively generating informative training data.
Original Article
View Cached Full Text

Cached at: 05/20/26, 08:36 AM 

# PROWL: Prioritized Regret-Driven Optimization for World Model Learning
Source: [https://arxiv.org/abs/2605.18803](https://arxiv.org/abs/2605.18803)
[View PDF](https://arxiv.org/pdf/2605.18803)[HTML \(experimental\)](https://arxiv.org/html/2605.18803v1)

> Abstract:Modern action\-conditioned video world models achieve strong short\-horizon visual realism, yet remain unreliable on rare, interaction\-critical transitions that dominate downstream planning and policy performance\. Because passive demonstration data systematically under\-samples these high\-impact regimes, improving robustness requires actively eliciting model failures rather than relying on their natural occurrence\. We introduce a KL\-constrained adversarial curriculum in which a policy is trained to expose high\-error trajectories of a diffusion\-based world model while remaining close to the behavior distribution\. The world model is continuously fine\-tuned on these adversarially discovered trajectories, yielding an adversarial training loop that converts rare failures into a stable, near\-distribution training signal without drifting into out\-of\-distribution exploitation\. To maintain pressure on unresolved weaknesses as the model improves, we propose a Prioritized Adversarial Trajectory \(PAT\) buffer that re\-ranks trajectories based on prediction error, action fidelity, and learning progress, focusing training on unresolved failure modes rather than repeatedly revisiting solved cases\. We implement our approach in the MineRL framework and evaluate it on held\-out out\-of\-distribution trajectories; PROWL improves robustness over models trained on passive data alone, reveals reward\-hacking behaviors under weak behavioral constraints, and demonstrates that effective adversarial world\-model training critically depends on balancing exploratory failure discovery with explicit behavioral regularization\. Our results suggest that scalable world models benefit not only from larger datasets, but also from selectively generating informative training data\.

## Submission history

From: Ahmet Hamdi Güzel Mr \[[view email](https://arxiv.org/show-email/4a677da0/2605.18803)\] **\[v1\]**Mon, 11 May 2026 14:24:19 UTC \(6,352 KB\)

Similar Articles

Scaling Automatic Research Agents via World Models

arXiv cs.LG

This paper identifies a scalability bottleneck in RL-trained automatic research agents—environment execution dominates training cost—and proposes World Model RL (WMRL) with online debiasing and inverse-variance denoising to replace real execution, achieving 3–4x training speedups and better generalization.

Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

arXiv cs.AI

This paper introduces DROPJ, a human-centred method for safely training and deploying agent policies by learning a world model from real-world trajectories, then eliciting human preferences with justifications to train a reward model for model predictive control. Experiments show that using human-generated simulated trajectories and justifications improves safety and reduces computational cost.