Agent as Policy for Robotic Manipulation
Summary
A general-purpose agent directly controls a physical robot by interpreting visuals, writing executable programs, and revising actions based on physical feedback across diverse manipulation tasks, achieving high success rates without task-specific training.
View Cached Full Text
Cached at: 09/15/26, 06:39 AM
Paper page - Agent as Policy for Robotic Manipulation
Source: https://huggingface.co/papers/2609.12541
Abstract
A general-purpose agent directly controls a physical robot by interpreting visuals, writing executable programs, and revising actions based on physical feedback across diverse manipulation tasks.
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduceAgent as Policy(AGP), which places task planning and execution under the agent’s control. Given a task and a robot interface, the agent interprets visual evidence, writesexecutable programs, issuesmotion commands, and revises its actions in response tophysical outcomes. This brings the agent’s reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiplereal-world manipulationtasks spanningprecision manipulation, dynamic motions, anddeformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, andbimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act asrobotic policies, extending their autonomy to physical manipulation throughruntime reasoning, programming, and interaction.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2609\.12541
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.12541 in a model README.md to link it from this page.
Datasets citing this paper1
#### Agent-as-Policy/agent-as-policy Viewer• Updatedabout 1 hour ago • 162 • 4 • 2
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.12541 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Agent as Policy for Robotic Manipulation
This paper introduces Agent as Policy (AGP), a framework that enables general-purpose agents to directly control physical robots for manipulation tasks without task-specific training, achieving high success rates across various real-world scenarios.
Robot-use agents
This article discusses the emerging trend of using LLMs as robot-control agents, akin to computer-use agents, and explores the potential for rapid diffusion of intelligence across the robotic ecosystem through cloud-based AI.
AgentOS
AgentOS provides a unified control layer for managing AI agents, tasks, and workspaces.
ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop
ProcAgent is a fully on-device, agentic, vision-based procedural assistant that uses a propose-and-verify architecture for real-time adaptive guidance on an NVIDIA Jetson AGX Orin. It supports both reactive and proactive modes with human-in-the-loop confirmation, achieving responsive interaction and positive user study ratings.
RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory
RoboArchitect automates the design of robotic agent harnesses by using a coding model to iteratively revise control code around a frozen policy, boosting success on unseen scenes from 9.7% to 43.3% and on unseen tasks from 12.7% to 40.2% without any model weight updates, while also evolving persistent memory across episodes.