NeuROK: Generative 4D Neural Object Kinematics

Hugging Face Daily Papers Papers
4d-dynamics neural-simulation object-kinematics transformer latent-space computer-vision generative-model

Summary

This paper introduces NeuROK, a data-driven approach for generative 4D neural object kinematics that learns a latent space and transformer-based encoder-decoder to simulate realistic temporal deformations of static objects under various physical conditions, overcoming limitations of predefined physical models.

Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative 4D dynamics -- realistic temporal deformations of static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization for object-centric physical systems. Specifically, we learn both a latent space representing all possible states of the object and a decoder that maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensional latent space from the Lagrangian mechanics' perspective in classical physics. We demonstrate the effectiveness and generality of this neural simulation framework across diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok
Original Article
View Cached Full Text

Cached at: 05/29/26, 02:59 AM

Paper page - NeuROK: Generative 4D Neural Object Kinematics

Source: https://huggingface.co/papers/2605.30347

Abstract

Neural Object Kinematics learns a data-driven parameterization for 4D dynamic object simulation by combining latent space representation with transformer-based encoding-decoding, enabling realistic temporal deformations across diverse object types.

Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative4D dynamics-- realistictemporal deformationsof static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization forobject-centric physical systems. Specifically, we learn both alatent spacerepresenting all possible states of the object and adecoderthat maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn atransformer-based encoder-decoder modelon acurated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensionallatent spacefrom theLagrangian mechanics’ perspective in classical physics. We demonstrate the effectiveness and generality of thisneural simulation frameworkacross diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2605\.30347

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.30347 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.30347 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.30347 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

D4RT: Teaching AI to see the world in four dimensions

Google DeepMind Blog

DeepMind introduces D4RT, a unified AI model for dynamic 4D scene reconstruction and tracking that is up to 300x more efficient than previous methods. The model uses a query-based Transformer architecture to solve complex spatial and temporal tasks for robotics and AR applications.

Domain randomization and generative models for robotic grasping

OpenAI Blog

Researchers explore a data generation pipeline using domain randomization and procedurally generated objects to train a deep neural network for robotic grasp planning. The proposed autoregressive model achieves >90% success on unseen objects in simulation and 80% in the real world, despite being trained only on random simulated objects.

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Hugging Face Daily Papers

RynnWorld-4D is a generative world model that co-produces future RGB, depth, and optical flow from a single RGB-D image and language instruction using a unified diffusion process, enabling efficient robotic manipulation through inverse dynamics policy learning. It achieves state-of-the-art on real-world bimanual manipulation tasks.