Transferring the Intelligence of VLMs to Robotic Control

Hugging Face Daily Papers Papers
vision-language-models robotic-control transfer-learning in-context-learning closed-loop-system benchmark-evaluation real-world-robotics

Summary

This paper presents RoboDawn, a method to transfer Vision-Language Model intelligence to robotic control, achieving state-of-the-art results on benchmarks with zero-shot and one-shot learning and successful real-world applications.

Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.
Original Article
View Cached Full Text

Cached at: 09/22/26, 03:25 AM

Paper page - Transferring the Intelligence of VLMs to Robotic Control

Source: https://huggingface.co/papers/2609.22966

Abstract

Humanscanseamlesslyadapttobothphysicalanddigitalworlds,suggestingthatwhileadigital-to-realgapexistsinembodiment,environmentandtask,humanintelligenceitselfmaytransferacrossthisgap.Thisnaturallyraisesafundamentalquestion:cantheintelligenceofvision-languagemodels(VLMs)similarlygeneralizefromthedigitalworldtothephysicalworldforroboticcontrol?WeinvestigatethisquestionthroughRoboDawn,ahuman-intuitiveinterfacethatexposesroboticcontroltoanagenticVLMthroughacompactsetofdiscretetranslation,rotation,andgrippercommands.Usingthisinterface,theVLMcontrolsarobotinaclosedloop:itobservesthecurrentvisualstate,reasonsaboutthenextaction,executesit,andadaptssubsequentdecisionstotheresultingstate.Furthermore,weintroduceanin-contextlearning(ICL)schemethatusesafewdemonstrationstogroundtheVLMinbothinterfaceusageandtask-solvingstrategies.ExperimentsonRoboTwin2.0C2RandRoboDojodemonstratethatRoboDawnachievesstrongperformancewithouttask-specificrobottraining.Inthezero-shotsetting,RoboDawnoutperformsseveralstrongpoliciestrainedonbenchmarkspecificrobotdata,whileasinglein-contextdemonstrationfurtheryieldssubstantialperformancegainsandestablishesstate-of-the-art(SOTA)results.OnRoboTwin2.0C2R,thesuccessrateincreasesfrom53.2%zero-shotto73.6%one-shot,exceedingthesolidbaselineπ0.5(46.0%).SimilargainsareobservedonRoboDojo,wheresuccessrateimprovesfrom35.67%zero-shotto47.17%one-shot.Thesameframeworkalsotransferstoreal-worldrobots,performingblock-in-basketandblockstackingonFranka.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.22966

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.22966 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.22966 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.22966 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Robots Need More than VLA and World Models

Hugging Face Daily Papers

This position paper argues that advancing robot intelligence requires integrating unstructured behavioral data through specialized interfaces for labeling, embodiment mapping, world modeling, and reward inference, rather than relying solely on scaling Vision-Language-Action (VLA) models and world models.