NVIDIA Enables the Next Era Of Physical AI Research With Agent Skills For Autonomous Vehicles, Robotics And Vision AI

<div id="bsf_rt_marker"></div><p><span style="font-weight: 400;">At CVPR, NVIDIA is unveiling new physical AI agent skills that </span><a href="https://blogs.nvidia.com/blog/cvpr-research-grasping-driving-agent-training/"><span style="font-weight: 400;">help researchers and developers</span></a><span style="font-weight: 400;"> speed the development of </span><a target="_blank" href="https://www.nvidia.com/en-us/solutions/autonomous-vehicles/"><span style="font-weight: 400;">autonomous vehicles</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://www.nvidia.com/en-us/industries/robotics/"><span style="font-weight: 400;">robots</span></a><span style="font-weight: 400;"> and </span><a target="_blank" href="https://www.nvidia.com/en-us/autonomous-machines/intelligent-video-analytics-platform/"><span style="font-weight: 400;">vision AI systems</span></a><span style="font-weight: 400;">.</span></p> <p><span style="font-weight: 400;">The core challenge in </span><a target="_blank" href="https://www.nvidia.com/en-us/glossary/generative-physical-ai/"><span style="font-weight: 400;">physical AI</span></a><span style="font-weight: 400;"> research isn’t simply developing stronger models. It’s building a full workflow around them — reconstructing real-world scenes, generating edge-case scenarios, training policies, evaluating behavior and rapidly iterating. Today, these steps are fragmented across separate tools, slowing the pace of experimentation as researchers struggle to piece them together.</span></p> <p><span style="font-weight: 400;">Earlier this week, NVIDIA announced </span><a target="_blank" href="https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-3-the-open-frontier-foundation-model-for-physical-ai"><span style="font-weight: 400;">NVIDIA Cosmos 3</span></a><span style="font-weight: 400;">, the open frontier model for physical AI and the world’s first full omnimodel unifying vision reasoning, world and action generation. Leading across the open model public leaderboards central to physical AI, the world foundation model provides core capabilities for physical AI development. </span><a target="_blank" href="https://github.com/NVIDIA/skills"><span style="font-weight: 400;">NVIDIA physical AI skills</span></a><span style="font-weight: 400;"> pair with Cosmos,  NVIDIA libraries and simulation frameworks to help researchers move from model capabilities to scalable end-to-end workflows faster than ever. </span></p> <h2><b>Advancing Autonomous Vehicle Research Beyond Recorded Miles</b></h2> <p><span style="font-weight: 400;">For AV researchers, the problem is the “long tail” of driving — rare interactions, unusual road geometry, lighting changes and edge-case behaviors that are difficult to repeatedly collect, but critical for training and validation.</span><span style="font-weight: 400;"><br /> </span></p> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-3" width="1200" height="675" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/NeuralReconstructionDemo.mp4?_=3" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/NeuralReconstructionDemo.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/NeuralReconstructionDemo.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">Neural Reconstruction skill demo in OpenClaw, showing a video re-rendered from an elevated virtual sensor viewpoint.</span></em></p> <p><span style="font-weight: 400;">With NVIDIA autonomous vehicle skills, researchers and developers can task AI agents to automate workflows for scene reconstruction from fleet data and generate synthetic scenarios. </span><a target="_blank" href="https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-neural-reconstruction"><span style="font-weight: 400;">Neural Reconstruction</span></a> <span style="font-weight: 400;">skills help AI agents turn fleet-captured data into editable 3D scenes for </span><a target="_blank" href="https://www.nvidia.com/en-us/solutions/autonomous-vehicles/simulation/"><span style="font-weight: 400;">simulation</span></a><span style="font-weight: 400;"> and synthetic data generation, while technologies including </span><a target="_blank" href="https://developer.nvidia.com/omniverse/nurec"><span style="font-weight: 400;">NVIDIA Omniverse NuRec</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://github.com/NVIDIA/instant-nurec"><span style="font-weight: 400;">InstantNuRec</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="http://www.github.com/NVIDIA/harmonizer"><span style="font-weight: 400;">Harmonizer</span></a><span style="font-weight: 400;"> and </span><a target="_blank" href="https://research.nvidia.com/labs/sil/projects/higs/"><span style="font-weight: 400;">HiGS accelerated renderer</span></a><span style="font-weight: 400;"> help accelerate reconstruction, improve scene realism and generate new views.</span></p> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-4" width="1200" height="340" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/InstantNuRec.mp4?_=4" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/InstantNuRec.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/InstantNuRec.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">InstantNuRec enables fast 3D Gaussian road-scene reconstruction from images without per-scene optimization.</span></em></p> <p><span style="font-weight: 400;">For AV researchers, repeatable simulation helps vary conditions, compare system responses and uncover failure modes across scenarios beyond what can be captured in real-world data. </span></p> <p><a target="_blank" href="https://huggingface.co/blog/drmapavone/nvidia-alpamayo-2"><span style="font-weight: 400;">NVIDIA AlpaGym</span></a><span style="font-weight: 400;">, an open source closed-loop reinforcement learning framework, extends that approach by connecting policy rollouts and high-fidelity simulation with agent skills, scaling across thousands of GPUs, to help researchers move through setup, rollout and evaluation. </span><a target="_blank" href="https://huggingface.co/nvidia/omni-dreams-models"><span style="font-weight: 400;">NVIDIA OmniDreams</span></a><span style="font-weight: 400;">, an action-conditioned generative world model, adds photorealistic rendering to the simulation loop, generating camera frames that respond directly to policy actions in real time.</span></p> <p><span style="font-weight: 400;">NVIDIA is also advancing AV research with its most powerful open driving foundation model to date: </span><a target="_blank" href="https://nvidianews.nvidia.com/news/nvidia-alpamayo-2-super-robotaxis"><span style="font-weight: 400;">NVIDIA Alpamayo 2 Super</span></a><span style="font-weight: 400;">, an open 32-billion-parameter reasoning vision language action (VLA) model that reasons, plans and acts across the full driving stack for safer, scalable level 4 development and deployment. </span></p> <h2><b>Advancing Vision AI Systems for the Real World</b></h2> <p><span style="font-weight: 400;">For vision AI research, the bottleneck is creating enough controlled examples to study how models behave when visual conditions, object states or temporal events change. Work in zero-shot anomaly detection, synthetic anomaly generation and few-shot defect recognition all run into the same data wall.</span></p> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-5" width="1200" height="675" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/Delta-Defect-Image-Generation.mp4?_=5" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/Delta-Defect-Image-Generation.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/Delta-Defect-Image-Generation.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">New skills for visual inspection generates multiple rare defects on different surfaces.</span></em></p> <p><a target="_blank" href="https://developer.nvidia.com/metropolis"><span style="font-weight: 400;">New NVIDIA Metropolis skills</span></a> <span style="font-weight: 400;">are helping researchers and developers use AI agents to generate synthetic visual scenarios, including anomalies, augment data and support pseudo-labeling. These skills benefit from Cosmos 3’s mixture-of-transformers architecture, which uses a reasoning transformer to analyze observations and feed instructions to a generation tower, helping scale physically grounded virtual worlds.</span></p> <p><span style="font-weight: 400;">Researchers building high-accuracy visual inspection models can use the </span><a target="_blank" href="https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-defect-image-generation"><span style="font-weight: 400;">Defect Image Generation skill</span></a><span style="font-weight: 400;"> to create examples of different defects across different surfaces using real images. The workflow combines NVIDIA Isaac Sim for simulation, Cosmos 3 and </span><a target="_blank" href="https://developer.nvidia.com/osmo"><span style="font-weight: 400;">NVIDIA OSMO </span></a><span style="font-weight: 400;">for orchestration and vision language reasoning — letting researchers create rare visual cases and assess whether models respond correctly.</span></p> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-6" width="1200" height="675" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/VSS3_Demo.mp4?_=6" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/VSS3_Demo.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/VSS3_Demo.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">New NVIDIA Metropolis VSS Blueprint skills extract insights from massive volumes of video data.</span></em></p> <p><span style="font-weight: 400;">For video AI agents, the </span><a target="_blank" href="https://build.nvidia.com/nvidia/video-search-and-summarization"><span style="font-weight: 400;">NVIDIA Metropolis Blueprint for video search and summarization (VSS)</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://developer.nvidia.com/tao-toolkit"><span style="font-weight: 400;">NVIDIA TAO</span></a><span style="font-weight: 400;"> and </span><a target="_blank" href="https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-video-data-augmentation"><span style="font-weight: 400;">Video Augmentation skills</span></a><span style="font-weight: 400;"> help extract insights from massive volumes of video data, fine-tune models and</span> <span style="font-weight: 400;">automate the build-and-evaluate loop. This gives researchers a more repeatable way to develop reasoning vision AI agents that can detect events, reason over complex scenes, summarize activity and send alerts.</span></p> <h2><b>Scaling Robot Learning With Agent-Ready Simulation Workflows</b></h2> <p><span style="font-weight: 400;">Teaching robots skills like navigating or manipulating comes down to iteration. For researchers, the bottleneck is building enough controlled environments and policy rollouts to understand how robot behavior changes across tasks, settings and embodiments — work that typically means stitching together simulation environments, task variations, policy training and evaluation by hand.</span><span style="font-weight: 400;"><br /> </span></p> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-7" width="1200" height="675" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/Isaac-Sim.mp4?_=7" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/Isaac-Sim.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/Isaac-Sim.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">NVIDIA Isaac Sim 6.0 includes agent-friendly skills and connectors to help automate workflows.</span></em></p> <p><span style="font-weight: 400;">With NVIDIA robotics skills, researchers can task AI agents to automate most common development steps across scene preparation, simulation and robot learning with </span><a target="_blank" href="https://developer.nvidia.com/omniverse"><span style="font-weight: 400;">NVIDIA Omniverse libraries</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://developer.nvidia.com/isaac/sim"><span style="font-weight: 400;">Isaac Sim</span></a><span style="font-weight: 400;"> and </span><a target="_blank" href="https://developer.nvidia.com/isaac/lab"><span style="font-weight: 400;">Isaac Lab</span></a><span style="font-weight: 400;"> frameworks. Agents can help launch simulation sessions, author scenes, control simulation, capture data and validate environments in Isaac Sim, while Isaac Lab skills support reinforcement learning setup, training, evaluation and custom environment development.</span></p> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-8" width="1200" height="675" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/COMPASS-grid.mp4?_=8" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/COMPASS-grid.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/COMPASS-grid.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">New NVIDIA Isaac mobility skills automate navigation workflows.</span></em></p> <p><span style="font-weight: 400;">Specialized skills extend that workflow to mobility and manipulation. </span><a target="_blank" href="https://github.com/NVlabs/COMPASS"><span style="font-weight: 400;">Isaac mobility skills</span></a><span style="font-weight: 400;"> support navigation workflows spanning scene search, USD conversion, environment registration, residual reinforcement learning and policy evaluation, while specialized Isaac Lab agentic workflows help with sim-to-sim and sim-to-real tasks such as environment building, physics tuning, debugging and profiling.</span></p> <p><span style="font-weight: 400;">For healthcare robotics, </span><a target="_blank" href="https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator"><span style="font-weight: 400;">Cosmos-H-Surgical-Simulator </span></a><span style="font-weight: 400;">advances research by generating realistic surgical robotics data for policy training and evaluation. By learning directly from real surgical data rather than hand-engineered physics models, it helps reduce the sim-to-real gap, supporting the development of autonomous surgical tasks.</span></p> <p><span style="font-weight: 400;">Cosmos 3 can further help generate synthetic data and scene variations, then support post-training with embodiment-specific behavior and environment data for tasks ranging from pick-and-place to dexterous manipulation.</span></p> <h2><b>NVIDIA Research at CVPR</b></h2> <p><span style="font-weight: 400;">NVIDIA technologies — including GPUs, open models, simulation frameworks and CUDA-accelerated libraries — were referenced in the majority of accepted CVPR 2026 papers, with adoption across leading global research labs and institutions including </span><span style="font-weight: 400;">Carnegie Mellon</span> <span style="font-weight: 400;">University</span><span style="font-weight: 400;">, </span><span style="font-weight: 400;">Stanford University</span><span style="font-weight: 400;">, </span><span style="font-weight: 400;">UC Berkeley</span><span style="font-weight: 400;">,</span> <span style="font-weight: 400;">Tsinghua University</span><span style="font-weight: 400;"> and </span><span style="font-weight: 400;">Peking University</span><span style="font-weight: 400;">.</span></p> <p><span style="font-weight: 400;">NVIDIA researchers are presenting work across computer vision, physical AI, autonomous systems, neural rendering, generative AI and robotics at </span><a target="_blank" href="https://www.nvidia.com/en-us/events/cvpr/"><span style="font-weight: 400;">CVPR</span></a><span style="font-weight: 400;">, running June 3-7 in Denver. </span></p> <p><span style="font-weight: 400;">NVIDIA’s CVPR presence also includes open research challenges that help benchmark progress in physical AI:</span></p> <ul> <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The </span><a target="_blank" href="https://www.aicitychallenge.org/"><span style="font-weight: 400;">AI City Challenge</span></a><span style="font-weight: 400;">, a premier computer vision competition for smart city applications  now in its tenth year.</span></li> <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The </span><a target="_blank" href="https://huggingface.co/spaces/nvidia/PhysicalAI-AV-OOD-Reasoning-Challenge-2026"><span style="font-weight: 400;">PAI-AV Reasoning Challenge</span></a><span style="font-weight: 400;">, a new open benchmark evaluating how well VLA models explain driving decisions using chain-of-causation labels.</span></li> <li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">The </span><a target="_blank" href="https://huggingface.co/spaces/nvidia/AlpasimE2EClosedLoopChallenge2026"><span style="font-weight: 400;">AlpaSim Closed-Loop End-to-End Driving Challenge</span></a><span style="font-weight: 400;">, a new open benchmark testing autonomous driving policies in closed-loop simulation on real-world reconstructed scenarios. </span></li> </ul> <div style="width: 1200px;" class="wp-video"><video class="wp-video-shortcode" id="video-93956-9" width="1200" height="675" preload="metadata" controls="controls"><source type="video/mp4" src="https://blogs.nvidia.com/wp-content/uploads/2026/06/RoboSim-Grid.mp4?_=9" /><a href="https://blogs.nvidia.com/wp-content/uploads/2026/06/RoboSim-Grid.mp4">https://blogs.nvidia.com/wp-content/uploads/2026/06/RoboSim-Grid.mp4</a></video></div> <p>&nbsp;</p> <p style="text-align: center;"><em><span style="font-weight: 400;">Grid of samples videos from new Robot Sim Dataset as a part of Cosmos 3 dataset release.</span></em></p> <p><span style="font-weight: 400;">NVIDIA is also expanding the research infrastructure behind physical AI with datasets for training, fine-tuning and evaluation. The </span><a target="_blank" href="https://huggingface.co/collections/nvidia/physical-ai"><span style="font-weight: 400;">NVIDIA Physical AI Dataset</span></a><span style="font-weight: 400;"> has surpassed 15 million+ downloads on </span><span style="font-weight: 400;">Hugging Face</span><span style="font-weight: 400;">, while </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim"><span style="font-weight: 400;">NVIDIA Isaac GR00T X Embodiment Sim</span></a><span style="font-weight: 400;"> has become one of the most-downloaded robotics datasets. New dataset releases include </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL"><span style="font-weight: 400;">GRAIL</span></a><span style="font-weight: 400;">, including roughly 50 hours of humanoid-object interaction data, and six synthetic video datasets used to train Cosmos 3 across </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Embodied-Robot-Scenes"><span style="font-weight: 400;">robotics</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Physical-Interaction-Scenes"><span style="font-weight: 400;">physics</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Digital-Human-Scenes"><span style="font-weight: 400;">digital humans</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Autonomous-Driving-Scenarios"><span style="font-weight: 400;">autonomous driving</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Warehouse-Operations-Scenes"><span style="font-weight: 400;">warehouse safety</span></a><span style="font-weight: 400;"> and </span><a target="_blank" href="https://huggingface.co/datasets/nvidia/PhysicalAI-WorldModel-Synthetic-Spatial-Reasoning"><span style="font-weight: 400;">spatial reasoning</span></a><span style="font-weight: 400;">.</span></p> <h2><b>Availability</b></h2> <p><span style="font-weight: 400;">NVIDIA physical AI agent tools and skills are now </span><a target="_blank" href="https://github.com/NVIDIA/skills"><span style="font-weight: 400;">openly available through GitHub</span></a><span style="font-weight: 400;">.</span><span style="font-weight: 400;"><br /> </span></p> <p><span style="font-weight: 400;">Agent skills and tools for synthetic data generation — </span><a target="_blank" href="https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-neural-reconstruction"><span style="font-weight: 400;">Neural Reconstruction</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-video-data-augmentation"><span style="font-weight: 400;">Video Augmentation</span></a><span style="font-weight: 400;">, </span><a target="_blank" href="https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-defect-image-generation"><span style="font-weight: 400;">Defect Image Generation</span></a><span style="font-weight: 400;"> — are also available to try instantly on NVIDIA Brev as </span><a target="_blank" href="https://brev.nvidia.com/physical-ai"><span style="font-weight: 400;">Physical AI Launchables</span></a><span style="font-weight: 400;">, preconfigured environments that bundle agent skills and tools for faster synthetic data generation and evaluation. Launchables run on hosted NVIDIA H100 Tensor Core GPUs and include free trial credits for researchers.</span></p> <p><i><span style="font-weight: 400;">Learn more about </span></i><a target="_blank" href="https://www.nvidia.com/en-us/events/cvpr/"><i><span style="font-weight: 400;">NVIDIA at CVPR</span></i></a><i><span style="font-weight: 400;"> and </span></i><a target="_blank" href="https://research.nvidia.com"><i><span style="font-weight: 400;">explore NVIDIA Research</span></i></a><i><span style="font-weight: 400;">’s work in physical AI, computer vision and autonomous systems. Get started with </span></i><a target="_blank" href="https://developer.nvidia.com/isaac"><i><span style="font-weight: 400;">Isaac GR00T and NVIDIA robotics tools</span></i></a><i><span style="font-weight: 400;">. </span></i></p>