SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
Summary
SimFoundry is a modular system that automates real-to-sim scene construction from video, generating digital twins and affordance-preserving variations for zero-shot robot policy training, achieving strong transfer to real-world tasks and high simulation-to-real performance prediction.
View Cached Full Text
Cached at: 06/29/26, 02:00 AM
Paper page - SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
Source: https://huggingface.co/papers/2606.28276 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
SimFoundry enables zero-shot real-world robot policy training through automated simulation construction and diverse scene variations that improve generalization and performance prediction.
Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shotreal-to-sim scene constructionfrom a video. SimFoundry generates sim-readydigital twinsand supports object, scene, and task editing, enabling the automated generation of diversedigital cousins:affordance-preserving variationsof reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and itsdigital cousins(variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundrysimulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.28276
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.28276 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.28276 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.28276 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
This paper proposes a strategic robustness objective for learning simulators in model-based reinforcement learning, formulated as a minimax game between a model player and an adversarial policy player. Theoretical guarantees and a provably convergent algorithm are provided, with experiments showing reduced prediction error and improved real-world policy transfer.
AI agents create virtual playgrounds to help robots get crucial training data
MIT CSAIL and Toyota Research Institute introduce SceneSmith, a system using AI agents powered by GPT-5.2 to automatically generate rich 3D virtual scenes, providing diverse simulation environments for robot training without extensive real-world testing.
RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures
RoboTALES introduces a two-stage framework combining LLM-based planning and VLM-based criticism to improve task-aligned video generation and robotic policy training, significantly outperforming existing methods on long-horizon manipulation tasks.
SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
SimWorld Studio is an open-source platform that uses an evolving coding agent to automatically generate and refine 3D environments for embodied agent learning. It leverages self-evolution and co-evolution mechanisms to create adaptive training scenarios, significantly improving agent performance.
SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
SimWorlds is a multi-agent framework that generates dynamic, editable 4D scenes from natural language, using Blender-specific procedural knowledge and a planner-coder-reviewer workflow, outperforming prior baselines.