Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
Summary
This paper proposes a strategic robustness objective for learning simulators in model-based reinforcement learning, formulated as a minimax game between a model player and an adversarial policy player. Theoretical guarantees and a provably convergent algorithm are provided, with experiments showing reduced prediction error and improved real-world policy transfer.
View Cached Full Text
Cached at: 05/29/26, 09:15 AM
# Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning Source: [https://arxiv.org/abs/2605.29032](https://arxiv.org/abs/2605.29032) [View PDF](https://arxiv.org/pdf/2605.29032) > Abstract:Model\-based reinforcement learning \(MBRL\) agents typically learn world models by minimizing predictive loss\. However, powerful RL optimizers inevitably exploit minor model inaccuracies, leading to simulator exploitation and a reality gap where policies succeed in simulation but fail in the real world\. We propose that the objective for learning simulators should be strategic robustness rather than predictive accuracy, and formulate this as a zero\-sum minimax game between a model player and an adversarial policy player\. We provide a comprehensive theoretical analysis: \(1\) an online learning guarantee showing the game is learnable with sublinear regret bounds; \(2\) a tractable critic\-based simplification bounding the global policy\-value gap by the local critic's loss; and \(3\) an Error\-MDP duality, proving that finding the worst\-case policy is formally dual to a standard RL problem where the reward is the one\-step critic error\. This duality yields a provably convergent active data selection algorithm\. Experiments on continuous control tasks demonstrate that our approach reduces prediction error in strategically important regions by $1\.5$\-$2\.2\\times$ and enables policies trained purely in simulation to match near\-optimal real\-world performance\. ## Submission history From: Christoph Dann \[[view email](https://arxiv.org/show-email/7ee4cd9e/2605.29032)\] **\[v1\]**Wed, 27 May 2026 19:31:37 UTC \(1,093 KB\)
Similar Articles
Mind the Sim-to-Real Gap & Think Like a Scientist
This paper studies when and how a planner should supplement a pre-trained simulator with real experiments in sequential decision problems, proposing Fisher-SEP to minimize posterior variance of a target policy's value.
SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
SimFoundry is a modular system that automates real-to-sim scene construction from video, generating digital twins and affordance-preserving variations for zero-shot robot policy training, achieving strong transfer to real-world tasks and high simulation-to-real performance prediction.
The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective
This paper formalizes the sim-to-real gap for foundation model agents as a Markov Decision Process problem, proposing a unified research agenda to adapt classical solutions like domain randomization for improving agent robustness and reliability in real-world deployment.
Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
This paper introduces DROPJ, a human-centred method for safely training and deploying agent policies by learning a world model from real-world trajectories, then eliciting human preferences with justifications to train a reward model for model predictive control. Experiments show that using human-generated simulated trajectories and justifications improves safety and reduces computational cost.
Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
Proposes ULPS, a framework integrating a calibrated LLM into RL training with uncertainty-modulated guidance and A*-based symbolic trajectories, achieving improved success rate and sample efficiency on MiniGrid-UnlockPickup.