SPEAR: A Simulator for Photorealistic Embodied AI Research

Hugging Face Daily Papers Papers
simulator embodied-ai photorealistic unreal-engine python-library rendering research

Summary

SPEAR is a Python library that controls Unreal Engine for photorealistic rendering at high speed, providing extensive programmability and unique ground-truth modalities for embodied AI research.

Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.
Original Article
View Cached Full Text

Cached at: 07/16/26, 05:44 PM

Paper page - SPEAR: A Simulator for Photorealistic Embodied AI Research

Source: https://huggingface.co/papers/2607.06701 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Interactivesimulatorshavebecomepowerfultoolsfortrainingembodiedagentsandgeneratingsyntheticvisualdata,butexistingphotorealisticsimulatorssufferfromlimitedgenerality,programmability,andrenderingspeed.WeaddresstheselimitationsbyintroducingSPEAR:ASimulatorforPhotorealisticEmbodiedAIResearch.Atitscore,SPEARisaPythonlibrarythatcanconnectto,andprogrammaticallycontrol,anyUnrealEngine(UE)applicationviaamodularpluginarchitecture.SPEARexposesover14KuniqueUEfunctionstoPython,representinganorder-of-magnitudeincreaseinprogrammablefunctionalityoverexistingUE-basedsimulators.Additionally,asingleSPEARinstancecanrender1920x1080photorealisticbeautyimagesdirectlyintoauser’sNumPyarrayat73framespersecond-anorderofmagnitudefasterthanexistingUEplugins-whilealsoprovidinggroundtruthimagemodalitiesthatarenotavailableinanyexistingUE-basedsimulator(e.g.,anon-diffuseintrinsicimagedecomposition,materialIDs,andphysicallybasedshadingparameters).Finally,SPEARintroducesanexpressivehigh-levelprogrammingmodelthatenablesuserstospecifycomplexgraphsofUEworkwitharbitrarydatadependenciesamongworkitems,andtoexecutethesegraphsdeterministicallywithinasingleUEframe.WedemonstratetheutilityofSPEARthroughadiversecollectionofexampleapplications:controllingmultipleembodiedagentswithdistinctactionspaces(e.g.,humans,cars,androbots)acrossseveralin-the-wildUEprojects;renderingphotorealisticcity-scaleenvironments;manipulatingUE’sproceduralcontentgenerationsystems;renderingsynchronizedmulti-viewimagesofdetailedhumanfaces;coordinatinganinteractiveco-simulationwiththeMuJoCophysicssimulator;andeditingsceneswithnaturallanguageviaanAIcodingassistant.

View arXiv pageView PDFProject pageGitHub529Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.06701 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.06701 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.06701 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

SPEAR: Code-Augmented Agentic Prompt Optimization

arXiv cs.CL

SPEAR is a code-augmented agentic prompt optimizer that uses a Python sandbox for structural error analysis, achieving state-of-the-art performance on multiple LLM evaluation suites including industrial judge tasks, BBH, and GSM8K.

AgentSPEX: An Agent SPecification and EXecution Language

Hugging Face Daily Papers

AgentSPEX introduces a domain-specific language for specifying modular, interpretable LLM-agent workflows with explicit control flow, state management, and a visual editor, outperforming existing Python-coupled frameworks.

ASPIRE: Agentic /Skills Discovery for Robotics

Hugging Face Daily Papers

ASPIRE is a continual learning system that autonomously develops and refines robot control programs through iterative exploration, achieving significant improvements in manipulation and household tasks while enabling sim-to-real transfer.