DragMesh-2: Physically Plausible Dexterous Hand-Object Interaction with Articulated Objects

Hugging Face Daily Papers Papers
dexterous-manipulation articulated-objects hand-object-interaction reinforcement-learning contact-aware simulation robotics

Summary

This paper presents DragMesh-2 and the PICA method for learning physically plausible dexterous hand-object interaction with articulated objects using contact-aware reinforcement learning in simulation, achieving effective manipulation without explicit tactile feedback.

Dexterous interaction with articulated objects is important for household, assistive, and humanoid manipulation, where multi-finger hands can provide compliant contact patterns beyond parallel-jaw grasping. However, articulated-object manipulation differs from static-object manipulation: the target part cannot be directly actuated, and its motion must emerge through sustained physical hand--handle contact. This makes the transition from object-centric articulated generation to hand-driven dexterous hand--object interaction non-trivial, since geometric trajectory replay or open-loop execution does not model the contact dynamics required to move the articulated part. Moreover, policies trained only for task completion under fixed dynamics can overfit nominal contact loads, especially without tactile or force feedback, and may degrade when the contact load changes. To address these challenges, we present DragMesh-2, a contact-driven framework for dexterous interaction with articulated objects that extends articulated interaction from object-centric generation to hand-driven dexterous hand--object interaction, where articulated motion must arise through physical contact. We further propose PICA, a physically informed contact-aware training mechanism that injects physical signals into policy learning without tactile or force feedback, improving robustness and task success under changing contact loads. Finally, we conduct systematic evaluation across multiple damping conditions and articulated-object categories to study robustness under contact-load variation, and provide a pure-geometry dexterous interaction resource to support future loco-manipulation and humanoid hand--object interaction research. Across seven GAPartNet objects, DragMesh-2 achieves stronger robustness under contact-load variation than the compared methods while maintaining high task success across damping conditions.
Original Article
View Cached Full Text

Cached at: 06/20/26, 02:28 PM

Paper page - DragMesh-2: Physically Plausible Dexterous Hand-Object Interaction with Articulated Objects

Source: https://huggingface.co/papers/2606.15133 Thanks for the question, and also thanks for sharing your blog.

In articulated object manipulation, the target joint cannot be directly actuated. The motion of a door or drawer has to come from sustained and effective physical contact between the hand and the handle. So our goal is not to have the policy simply replay a hand trajectory, but to make it learn how to drive the object through contact.

The intuition behind PICA comes from how people actually pull objects. Usually, we first establish contact and then adjust the next action based on the recent interaction. If the hand is still on the handle but the door or drawer is not moving enough, we may pull harder or change the pulling direction. If contact is lost, we first recover contact. PICA tries to encode this kind of short-term interaction feedback.

We do not use explicit tactile sensing or force feedback. Instead, we use hand-handle contact information available in the simulation. The policy receives a short contact-interaction history, so it can condition its action on whether contact has been maintained or lost, and whether the recent contact has produced articulation progress. In the PPO reward, we also include contact-aware terms such as detachment penalty, action saturation penalty, and task-progress reward. In addition, we introduce damping and contact-load variation during training, so the policy cannot simply overfit to one nominal dynamics setting.

So PICA injects physical interaction signals into policy learning through simulator-derived contact state and history, contact-aware reward shaping, and dynamics randomization. This encourages the policy to learn the behavior of maintaining contact and driving the target joint through contact, rather than only matching a geometric trajectory.

For real-world hardware generalization, I would view it as promising but not fully solved yet. One advantage of PICA is that it does not rely on explicit tactile or force feedback, so the formulation is closer to many real robot hand setups. However, real hardware still introduces friction and contact mismatch, actuator latency, calibration error, backlash, compliance, and noisy handle pose estimation. Also, the clean contact state and history available in the simulation would need to be estimated indirectly from vision, proprioception, motor current, or other signals in a real system.

Therefore, I see PICA more as a step toward hardware-feasible contact-aware learning, rather than a complete sim-to-real solution :)

Similar Articles

Zero-Shot Sim-to-Real Robot Learning: A Dexterous Manipulation Study on Reactive Catching

Hugging Face Daily Papers

This paper introduces Domain-Randomized Instance Set (DRIS), a method that simultaneously represents multiple randomized instances to improve sim-to-real transfer for dexterous manipulation. It demonstrates zero-shot transfer on a challenging reactive catching task with a flat plate end-effector, requiring no real-world fine-tuning.

Learning dexterity

OpenAI Blog

OpenAI announces Dactyl, a system that learns robotic hand dexterity through simulation and reinforcement learning, using LSTMs to generalize across different physical environments and the Rapid PPO implementation to train policies that transfer to real-world manipulation tasks.

Revisiting Articulated Parts Perception in Robot Manipulation

Hugging Face Daily Papers

This paper introduces Geometric Primary Structure (GPS), a new representation for articulated parts perception in robot manipulation, enabling efficient VR-based annotation and achieving a 73% success rate without fine-tuning.