Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping

# Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping
Source: [https://arxiv.org/html/2606.31200](https://arxiv.org/html/2606.31200)
Tao Chen1, Lizheng Liu1†, Jiaxu Wang1, Ziyue Jiang1, Ruiqi Tian2, JiGuang Huo1, Zhongxue Gan11Tao Chen, Lizheng Liu, Jiaxu Wang, Ziyue Jiang, JiGuang Huo, and Zhongxue Gan are with Fudan University, Shanghai, China \(e\-mail:lzliu@fudan\.edu\.cn\)\.2Ruiqi Tian is with Kean University\.†Corresponding author\.

###### Abstract

Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM\-based methods rely on visual similarity for object matching—neglecting physical affordances such as handle graspability and material fragility—and operate open\-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse\. We present Agentic RAG\-VLM, a unified framework that bridges VLM\-based semantic understanding and physically grounded grasp execution by integrating retrieval\-augmented generation \(RAG\) with vision\-language models \(VLMs\) and agentic self\-reflective planning\. Agentic RAG\-VLM introduces three tightly coupled components: \(1\) a Hierarchical Affordance\-Aware RAG \(HAA\-RAG\) that encodes four\-dimensional affordance descriptors—type, material, fragility, and graspable region—and retrieves strategies by functional affordance compatibility rather than visual appearance; \(2\) a Scene Graph Constraint Reasoner that constructs spatial relationship graphs from VLM perception and translates proximity, occlusion, and support constraints into concrete grasp parameter adjustments; and \(3\) an Agentic Self\-Reflective Pipeline with a 14\-type failure taxonomy and three\-level adaptive retry for closed\-loop grasp refinement\. Evaluated on a 12\-task benchmark spanning single\-grasp, interactive, and long\-horizon scenarios with 360 trials per configuration, Agentic RAG\-VLM achieves 78\.3% overall success—a 53\.3 pp absolute gain over VLM\-only baselines—demonstrating that affordance\-aware retrieval, scene graph reasoning, and agentic recovery are jointly essential for robust manipulation\.

## IIntroduction

Robotic grasping in unstructured environments requires integrating visual perception, semantic understanding, and physical execution\. A general\-purpose grasping pipeline typically involves perceiving the scene from sensor inputs, planning a grasp strategy, and executing it through a manipulator\. While each stage has seen significant progress independently—from learning\-based grasp detection\[[1](https://arxiv.org/html/2606.31200#bib.bib1),[2](https://arxiv.org/html/2606.31200#bib.bib2)\]to language\-conditioned imitation\[[3](https://arxiv.org/html/2606.31200#bib.bib3)\]—integrating them into a coherent system that handles diverse objects in cluttered, multi\-object environments remains largely unsolved\. This is especially hard for long\-horizon tasks like table clearing, where sequential dependencies and accumulated errors require both accurate planning and adaptive failure recovery\.

Recent vision\-language models \(VLMs\) have opened new directions for bridging perception and planning\. RT\-2\[[4](https://arxiv.org/html/2606.31200#bib.bib4)\]pioneers end\-to\-end VLM\-to\-action by tokenizing robotic actions alongside language; SayCan\[[5](https://arxiv.org/html/2606.31200#bib.bib5)\]decouples high\-level planning from low\-level execution through learned affordance functions; VoxPoser\[[6](https://arxiv.org/html/2606.31200#bib.bib6)\]and Code as Policies\[[7](https://arxiv.org/html/2606.31200#bib.bib7)\]generate 3D value maps and executable programs, respectively; and ManipLLM\[[8](https://arxiv.org/html/2606.31200#bib.bib8)\]fine\-tunes multimodal LLMs for 6\-DoF grasp prediction\. These advances point toward general\-purpose language\-conditioned manipulation\. Yet three gaps remain:

Semantic\-Manipulation Gap\.VLM\-based systems rely on visual similarity for object matching, but visual resemblance does not imply manipulation compatibility\. A ceramic mug and a glass vase may occupy nearby regions in CLIP\[[9](https://arxiv.org/html/2606.31200#bib.bib9)\]embedding space—both are cylindrical, hollow, and similarly sized—yet the mug should be grasped by its handle with a firm grip, while the vase requires a cautious side approach with reduced force\. This mismatch between visual similarity and manipulation compatibility\[[10](https://arxiv.org/html/2606.31200#bib.bib10)\]is largely unaddressed\.

Scene\-Unaware Planning\.Most systems reason about the target object in isolation\. In cluttered environments, a cup placed 5 cm from a fragile wine glass necessitates a cautious lateral approach with reduced force, whereas the same cup on an empty table can be grasped directly from above\. Scene graphs have been adopted for task and motion planning\[[11](https://arxiv.org/html/2606.31200#bib.bib11)\], but typically demand ground\-truth annotations and do not translate spatial reasoning into concrete grasp parameter adjustments\. SpatialVLM\[[12](https://arxiv.org/html/2606.31200#bib.bib12)\]endows VLMs with spatial understanding but does not connect it to manipulation constraints\.

Failure Recovery\.Classical grasp planners\[[1](https://arxiv.org/html/2606.31200#bib.bib1),[2](https://arxiv.org/html/2606.31200#bib.bib2),[13](https://arxiv.org/html/2606.31200#bib.bib13)\]operate open\-loop, generating grasp poses without learning from failures\. Inner Monologue\[[14](https://arxiv.org/html/2606.31200#bib.bib14)\]introduces VLM\-based feedback but provides only unstructured observations \(“the object slipped”\) without systematic classification\. Reflexion\[[15](https://arxiv.org/html/2606.31200#bib.bib15)\]demonstrates that verbal self\-reflection improves sequential decision\-making, yet applying reflection to physical manipulation requires translating linguistic feedback into concrete, physics\-grounded corrections—“the object slipped” must map to “increase grip force by 20%,” not merely a rephrased retry\.

To address these gaps, we propose Agentic RAG\-VLM \(Fig\.[1](https://arxiv.org/html/2606.31200#S1.F1)\), a unified framework integrating affordance\-aware experience retrieval, scene graph constraint reasoning, and agentic self\-reflective planning for robust robotic grasping\. The contributions are summarized as follows:

- •We propose Agentic RAG\-VLM, a unified framework that integrates affordance\-aware retrieval\-augmented generation with scene graph constraint reasoning and agentic self\-reflective planning, bridging the gap between VLM\-based semantic understanding and physically grounded grasp execution\.
- •The framework introduces Hierarchical Affordance\-Aware RAG \(HAA\-RAG\), a three\-level retrieval pipeline that matches grasp experiences by functional affordance rather than visual similarity, and a Scene Graph Constraint Reasoner that translates spatial relationships into concrete grasp parameter adjustments for safe manipulation in cluttered environments\.
- •An Agentic Self\-Reflective Pipeline that employs structured failure diagnosis with a 14\-type taxonomy and three\-level adaptive retry, enabling the system to learn from failed attempts and progressively refine grasp execution through physics\-grounded corrections\.

![Refer to caption](https://arxiv.org/html/2606.31200v1/figures/fig.1.png)Figure 1:Overview of Agentic RAG\-VLM\. The VLM\-based Task Planner processes multimodal input and orchestrates a seven\-stage execution pipeline: scene perception, affordance\-aware retrieval via HAA\-RAG, scene graph constraint reasoning \(Δ​𝐠\\Delta\\mathbf\{g\}\), and grasp execution\. Upon failure \(×\\times\), the recovery loop diagnoses failures via a 14\-type taxonomy, escalates corrections through three levels \(L1: parameter tuning→\\toL2: method switch→\\toL3: full replan\), and stores successful grasps in episodic memoryℳ\\mathcal\{M\}for cross\-task transfer\.
## IIRelated Work

VLMs for Manipulation\.Recent VLM\-based approaches range from end\-to\-end action tokenization \(RT\-2\[[4](https://arxiv.org/html/2606.31200#bib.bib4)\]\) to modular planning with affordance grounding \(SayCan\[[5](https://arxiv.org/html/2606.31200#bib.bib5)\]\), 3D value maps \(VoxPoser\[[6](https://arxiv.org/html/2606.31200#bib.bib6)\]\), code generation \(Code as Policies\[[7](https://arxiv.org/html/2606.31200#bib.bib7)\]\), and fine\-tuned grasp prediction \(ManipLLM\[[8](https://arxiv.org/html/2606.31200#bib.bib8)\], RoboDexVLM\[[16](https://arxiv.org/html/2606.31200#bib.bib16)\], GPT\-4V\[[17](https://arxiv.org/html/2606.31200#bib.bib17)\]\)\. However, these rely on parametric knowledge alone without retrieved experience or structured failure recovery\.

RAG and Affordance in Robotics\.RAG\[[18](https://arxiv.org/html/2606.31200#bib.bib18)\]reduces hallucination by grounding generation in retrieved documents, but its robotic application remains nascent\. REFLECT\[[19](https://arxiv.org/html/2606.31200#bib.bib19)\]summarizes failure experiences without integrating retrieval into planning\. Meanwhile, affordance reasoning\[[10](https://arxiv.org/html/2606.31200#bib.bib10)\]has been realized through pixel\-level maps\[[20](https://arxiv.org/html/2606.31200#bib.bib20)\]and part\-based analysis\[[21](https://arxiv.org/html/2606.31200#bib.bib21)\], but these predict*where*to grasp, not*how*—they identify graspable regions on an object surface without specifying the complete grasp strategy \(force, width, approach direction, grasp type\) appropriate for that object’s physical affordance\. Our HAA\-RAG bridges this gap by explicitly encoding four\-dimensional affordance descriptors \(type, material, fragility, graspable region\) and retrieving complete, physically grounded grasp strategies matched by functional affordance compatibility rather than visual similarity\.

Failure Recovery and Scene Reasoning\.Classical planners\[[1](https://arxiv.org/html/2606.31200#bib.bib1),[2](https://arxiv.org/html/2606.31200#bib.bib2),[13](https://arxiv.org/html/2606.31200#bib.bib13)\]operate open\-loop\. Inner Monologue\[[14](https://arxiv.org/html/2606.31200#bib.bib14)\]and Reflexion\[[15](https://arxiv.org/html/2606.31200#bib.bib15)\]introduce verbal feedback but lack physics\-grounded corrections for manipulation\. Scene graphs have been used for task planning\[[11](https://arxiv.org/html/2606.31200#bib.bib11)\]but typically require ground\-truth annotations\. SpatialVLM\[[12](https://arxiv.org/html/2606.31200#bib.bib12)\]adds spatial reasoning to VLMs without connecting it to grasp constraints\. Our pipeline addresses both gaps with a 14\-type failure taxonomy mapping to quantitative corrections and VLM\-constructed scene graphs that infer manipulation constraints without annotations\. In contrast to these approaches, Agentic RAG\-VLM uniquely integrates all five capabilities—affordance\-aware retrieval, RAG\-based experience grounding, scene graph constraint reasoning, structured failure recovery, and multi\-step agentic planning—within a single unified framework\.

## IIIAffordance\-Aware Retrieval and Constraint Reasoning

This section details how the system transforms a scene observation into a physically grounded grasp plan through two complementary stages: affordance\-aware experience retrieval \(Sec\.[III\-A](https://arxiv.org/html/2606.31200#S3.SS1)\) and spatial constraint reasoning \(Sec\.[III\-B](https://arxiv.org/html/2606.31200#S3.SS2)\)\. Given an RGB\-D observationIIand a natural language instructionqq, the system produces a grasp action𝐠=\(𝐩,𝐝,w,f,τ\)\\mathbf\{g\}=\(\\mathbf\{p\},\\mathbf\{d\},w,f,\\tau\)—position, approach direction, gripper aperture, grip force, and grasp type \(τ∈\{\\tau\\in\\\{power, pinch, side\}\\\}\)—refined through three modular stages: \(1\) affordance\-aware retrieval via HAA\-RAG, \(2\) scene graph constraint reasoning, and \(3\) agentic planning with self\-reflective retry \(Sec\.[IV](https://arxiv.org/html/2606.31200#S4)\)\.

### III\-AHierarchical Affordance\-Aware RAG

Standard RAG retrieves by visual similarity \(e\.g\., CLIP\[[9](https://arxiv.org/html/2606.31200#bib.bib9)\]embeddings\), but a bowl must be grasped along its rim while a cup uses its handle—visual similarity yields wrong strategies\. This is precisely the “neglecting object\-specific physical affordances” problem identified in Sec\.[I](https://arxiv.org/html/2606.31200#S1): visually similar objects \(e\.g\., a ceramic mug and a glass vase\) may require fundamentally different manipulation strategies due to their distinct affordance properties\. HAA\-RAG directly addresses this by matching experiences through*affordance compatibility*instead of visual resemblance \(Fig\.[2](https://arxiv.org/html/2606.31200#S3.F2)\)\.

The foundation of this approach is a curated knowledge base𝒦=\{e1,…,eN\}\\mathcal\{K\}=\\\{e\_\{1\},\\dots,e\_\{N\}\\\}withN=116N=116manipulation experiences, where each entryei=\(ci,𝐚i,𝐠i,yi,𝐯i\)e\_\{i\}=\(c\_\{i\},\\mathbf\{a\}\_\{i\},\\mathbf\{g\}\_\{i\},y\_\{i\},\\mathbf\{v\}\_\{i\}\)stores:

- •cic\_\{i\}: Object category \(e\.g\., “mug”, “bowl”, “screwdriver”\)
- •𝐚i=\(aitype,aimat,aifrag,aireg\)\\mathbf\{a\}\_\{i\}=\(a^\{\\mathrm\{type\}\}\_\{i\},a^\{\\mathrm\{mat\}\}\_\{i\},a^\{\\mathrm\{frag\}\}\_\{i\},a^\{\\mathrm\{reg\}\}\_\{i\}\): Affordance descriptor with primary type \(8 categories:Graspable\_Body/Handle/Edge,Pinchable,Wrappable,Fragile,Tool\_Handle,Clampable\), material, fragilityafrag∈\[0,1\]a^\{\\mathrm\{frag\}\}\\in\[0,1\], and graspable region
- •𝐠i\\mathbf\{g\}\_\{i\}: Grasp parameters that led to the outcome
- •yi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}: Binary success label
- •𝐯i\\mathbf\{v\}\_\{i\}: Visual feature embedding \(CLIP ViT\-L/14\)

Failed experiences \(yi=0y\_\{i\}=0\) are also retained with a contrastive penalty \(scores reduced by 30%\) to discourage repeating failed strategies while remaining available as negative examples for the reflection module \(Sec\.[IV\-B](https://arxiv.org/html/2606.31200#S4.SS2)\)\.

Leveraging this knowledge base, HAA\-RAG implements a three\-level coarse\-to\-fine retrieval pipeline\. In the first level, the VLM identifies the target object categorycqc\_\{q\}from instructionqqand the RGB image, retrieving the top\-k1=30k\_\{1\}=30candidates from𝒦\\mathcal\{K\}by category match: exact matches scorescat=1\.0s\_\{\\mathrm\{cat\}\}=1\.0, same\-superclass fuzzy matches \(e\.g\., “mug”→\\to“cup”\) score0\.50\.5, and unrelated categories are discarded\.

In the second level, surviving candidates are re\-scored by affordance similarity, capturing functional compatibility between the query object and stored experiences:

saff​\(𝐚q,𝐚i\)=0\.5⋅𝟙​\[aqtype=aitype\]⏟affordance type match\+0\.2⋅smat⏟material\+0\.15⋅\(1−\|Δ​f\|\)⏟fragility\+0\.15⋅sreg⏟region\\begin\{split\}s\_\{\\mathrm\{aff\}\}\(\\mathbf\{a\}\_\{q\},\\mathbf\{a\}\_\{i\}\)&=\\underbrace\{0\.5\\cdot\\mathbb\{1\}\[a\_\{q\}^\{\\mathrm\{type\}\}=a\_\{i\}^\{\\mathrm\{type\}\}\]\}\_\{\\text\{affordance type match\}\}\+\\underbrace\{0\.2\\cdot s\_\{\\mathrm\{mat\}\}\}\_\{\\text\{material\}\}\\\\ &\\quad\+\\underbrace\{0\.15\\cdot\(1\-\|\\Delta f\|\)\}\_\{\\text\{fragility\}\}\+\\underbrace\{0\.15\\cdot s\_\{\\mathrm\{reg\}\}\}\_\{\\text\{region\}\}\\end\{split\}\(1\)wheresmat=𝟙​\[aqmat=aimat\]s\_\{\\mathrm\{mat\}\}=\\mathbb\{1\}\[a\_\{q\}^\{\\mathrm\{mat\}\}=a\_\{i\}^\{\\mathrm\{mat\}\}\]is material compatibility \(e\.g\., both ceramic\),Δ​f=aqfrag−aifrag\\Delta f=a\_\{q\}^\{\\mathrm\{frag\}\}\-a\_\{i\}^\{\\mathrm\{frag\}\}captures fragility similarity, andsreg=𝟙​\[aqreg=aireg\]s\_\{\\mathrm\{reg\}\}=\\mathbb\{1\}\[a\_\{q\}^\{\\mathrm\{reg\}\}=a\_\{i\}^\{\\mathrm\{reg\}\}\]is graspable region overlap\. The dominant weight on affordance type \(0\.50\.5\) ensures that functional compatibility is the primary selection criterion\. Candidates scoring below thresholdτaff=0\.3\\tau\_\{\\mathrm\{aff\}\}=0\.3are discarded\.

Finally, the remaining candidates are re\-ranked by CLIP visual similaritysvis=cos​\(𝐯q,𝐯i\)s\_\{\\mathrm\{vis\}\}=\\mathrm\{cos\}\(\\mathbf\{v\}\_\{q\},\\mathbf\{v\}\_\{i\}\)using the current RGB observation’s embedding, ensuring that among affordance\-compatible experiences, the most visually similar one is selected for fine\-grained parameter matching\. The final retrieval score fuses all three levels \(Eq\.[1](https://arxiv.org/html/2606.31200#S3.E1)\) assfinal=0\.2​scat\+0\.4​saff\+0\.4​sviss\_\{\\mathrm\{final\}\}=0\.2\\,s\_\{\\mathrm\{cat\}\}\+0\.4\\,s\_\{\\mathrm\{aff\}\}\+0\.4\\,s\_\{\\mathrm\{vis\}\}, and the top\-k=3k=3experiences are returned to the planner as candidate strategies\.

![Refer to caption](https://arxiv.org/html/2606.31200v1/figures/fig.2.png)Figure 2:Hierarchical Affordance\-Aware RAG \(HAA\-RAG\) pipeline\. Experiences are progressively filtered through three levels: category matching, affordance scoring, and visual re\-ranking, reducing 116 candidates to 3 physically appropriate strategies\. Right annotations show candidate count at each stage\.
### III\-BScene Graph Constraint Reasoning

While HAA\-RAG determines*how*to grasp the target object based on its intrinsic affordance properties, real\-world manipulation must also account for the*extrinsic spatial context*imposed by neighboring objects\. The Scene Graph Constraint Reasoner addresses this by constructing a structured spatial representation and translating it into concrete grasp parameter adjustments\.

From the VLM’s scene analysis, we construct a directed graph𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\)where each nodevj∈𝒱v\_\{j\}\\in\\mathcal\{V\}encodes object attributes\(𝐩j,cj,statej,ajfrag\)\(\\mathbf\{p\}\_\{j\},c\_\{j\},\\mathrm\{state\}\_\{j\},a\_\{j\}^\{\\mathrm\{frag\}\}\)—position, category, physical state \(empty, filled, or stacked\), and fragility score—and each directed edgeej​k=\(vj,vk,rj​k\)∈ℰe\_\{jk\}=\(v\_\{j\},v\_\{k\},r\_\{jk\}\)\\in\\mathcal\{E\}captures a spatial relationrj​kr\_\{jk\}from 16 predefined types includingOn\_Top\_Of,Contains,Occludes,Supports,Adjacent\_To, andBehind\. A rule\-based Constraint Analyzer then traverses𝒢\\mathcal\{G\}starting from the target nodevtv\_\{t\}, examiningvtv\_\{t\}’s attributes and neighbors within a radius ofδcheck=0\.15\\delta\_\{\\mathrm\{check\}\}=0\.15m to infer active constraints𝒞vt\\mathcal\{C\}\_\{v\_\{t\}\}\. Four constraint types are defined, each mapping to specific parameter adjustments:

1. 1\.Content Preservation\(𝒞content\\mathcal\{C\}\_\{\\mathrm\{content\}\}\): For filled containers—constrains approach to vertical \(𝐝=\[0,0,−1\]\\mathbf\{d\}=\[0,0,\-1\]\) with slow velocity to prevent spillage\.
2. 2\.Collision Avoidance\(𝒞collision\\mathcal\{C\}\_\{\\mathrm\{collision\}\}\): When fragile neighborvfv\_\{f\}withinδsafe=0\.10\\delta\_\{\\mathrm\{safe\}\}=0\.10m—applies force reductionf′=0\.8​ff^\{\\prime\}=0\.8f, increases approach height by 30 mm, and biases approach direction away fromvfv\_\{f\}\.
3. 3\.Support Dependency\(𝒞support\\mathcal\{C\}\_\{\\mathrm\{support\}\}\): When aSupportsedge\(vt,vs\)\(v\_\{t\},v\_\{s\}\)exists, the system inserts a prerequisite action to remove the supported objectvsv\_\{s\}before attempting to graspvtv\_\{t\}\.
4. 4\.Occlusion Handling\(𝒞occlusion\\mathcal\{C\}\_\{\\mathrm\{occlusion\}\}\): When occluder blocks direct line\-of\-sight to target—modifies approach to lateral path that avoids the occluding obstacle\.

The inferred constraints are compiled into aconstraint adjustment vectorΔ​𝐠=\(Δ​𝐝,Δ​f,Δ​h\)\\Delta\\mathbf\{g\}=\(\\Delta\\mathbf\{d\},\\Delta f,\\Delta h\)that additively modifies the planned grasp action:𝐠′=𝐠⊕Δ​𝐠\\mathbf\{g\}^\{\\prime\}=\\mathbf\{g\}\\oplus\\Delta\\mathbf\{g\}\. Multiple active constraints are composed sequentially, with collision avoidance taking priority \(Fig\.[3](https://arxiv.org/html/2606.31200#S3.F3)\)\.

![Refer to caption](https://arxiv.org/html/2606.31200v1/figures/fig.3.png)Figure 3:Scene graph constraint reasoning\. Left: scene graph𝒢\\mathcal\{G\}with object nodes and spatial relations\. Right: four constraint types inferred from the graph, each mapped to concrete parameter adjustmentsΔ​𝐠\\Delta\\mathbf\{g\}that modify the planned grasp\.

## IVAgentic Self\-Reflective Grasp Planning

Real\-world execution is noisy, and initial failures are common\. Agentic RAG\-VLM employs an Agentic Self\-Reflective Pipeline providing*closed\-loop*execution with structured failure diagnosis and adaptive recovery\.

### IV\-AReAct\-Style Planning Loop

Inspired by the ReAct framework\[[22](https://arxiv.org/html/2606.31200#bib.bib22)\], the VLM iterates through structuredThought→\\rightarrowAction→\\rightarrowObservationcycles until either a successful grasp is achieved or the maximum retry budgetRmax=3R\_\{\\max\}=3is exhausted\.

Algorithm 1Agentic Self\-Reflective Grasp Planning0:RGB\-D image

II, instruction

qq, knowledge base

𝒦\\mathcal\{K\}, budget

RmaxR\_\{\\max\}
0:Grasp action

𝐠∗\\mathbf\{g\}^\{\*\}or failure report

1:

\{e1,…,ek\}←HAA\-RAG​\(I,q,𝒦\)\\\{e\_\{1\},\\dots,e\_\{k\}\\\}\\leftarrow\\textsc\{HAA\-RAG\}\(I,q,\\mathcal\{K\}\)\{Retrieve experiences\}

2:

𝒢←BuildSceneGraph​\(I\)\\mathcal\{G\}\\leftarrow\\textsc\{BuildSceneGraph\}\(I\)\{Construct scene graph\}

3:

𝒞←InferConstraints​\(𝒢,vt\)\\mathcal\{C\}\\leftarrow\\textsc\{InferConstraints\}\(\\mathcal\{G\},v\_\{t\}\)\{Infer constraints\}

4:

𝐠0←InitGrasp​\(e1,𝒞,ℳ\)\\mathbf\{g\}\_\{0\}\\leftarrow\\textsc\{InitGrasp\}\(e\_\{1\},\\mathcal\{C\},\\mathcal\{M\}\)\{Initial plan from memory/RAG\}

5:for

r=0r=0to

RmaxR\_\{\\max\}do

6:Thought:Reason about

𝐠r\\mathbf\{g\}\_\{r\},

𝒞\\mathcal\{C\}, and history

ℋ\\mathcal\{H\}
7:Action:Execute grasp

𝐠r\\mathbf\{g\}\_\{r\}
8:Observation:

\(yr,Qr,ϕr\)←Evaluate​\(𝐠r\)\(y\_\{r\},Q\_\{r\},\\boldsymbol\{\\phi\}\_\{r\}\)\\leftarrow\\textsc\{Evaluate\}\(\\mathbf\{g\}\_\{r\}\)
9:if

yr=Successy\_\{r\}=\\textsc\{Success\}then

10:

ℳ←ℳ∪\{\(cq,𝐠r\)\}\\mathcal\{M\}\\leftarrow\\mathcal\{M\}\\cup\\\{\(c\_\{q\},\\mathbf\{g\}\_\{r\}\)\\\}\{Store in episodic memory\}

11:return

𝐠r\\mathbf\{g\}\_\{r\}
12:endif

13:

ℱr←ClassifyFailure​\(ϕr\)\\mathcal\{F\}\_\{r\}\\leftarrow\\textsc\{ClassifyFailure\}\(\\boldsymbol\{\\phi\}\_\{r\}\)\{14\-type taxonomy\}

14:

Δ​𝐠r←GetCorrection​\(ℱr,r\)\\Delta\\mathbf\{g\}\_\{r\}\\leftarrow\\textsc\{GetCorrection\}\(\\mathcal\{F\}\_\{r\},r\)\{Level\-dependent\}

15:

𝐠r\+1←ApplyCorrection​\(𝐠r,Δ​𝐠r,r\)\\mathbf\{g\}\_\{r\+1\}\\leftarrow\\textsc\{ApplyCorrection\}\(\\mathbf\{g\}\_\{r\},\\Delta\\mathbf\{g\}\_\{r\},r\)
16:

ℋ←ℋ∪\{\(𝐠r,ℱr,Qr\)\}\\mathcal\{H\}\\leftarrow\\mathcal\{H\}\\cup\\\{\(\\mathbf\{g\}\_\{r\},\\mathcal\{F\}\_\{r\},Q\_\{r\}\)\\\}\{Update history\}

17:endfor

18:returnFailure

Algorithm[1](https://arxiv.org/html/2606.31200#alg1)formalizes the loop\. The initial grasp𝐠0\\mathbf\{g\}\_\{0\}is generated from the top\-ranked retrieved experience, adjusted by scene graph constraints and episodic memoryℳ\\mathcal\{M\}—a session\-level cache of successful grasps indexed by object category\. The retry budgetRmax=3R\_\{\\max\}=3balances recovery from failure modes against preventing indefinite loops\.

### IV\-BFailure Recovery with Structured Reflection

The Reflection Module generates*quantitative*corrections grounded in physics\. The seven quality factorsϕ=\(ϕ1,…,ϕ7\)\\boldsymbol\{\\phi\}=\(\\phi\_\{1\},\\dots,\\phi\_\{7\}\)from the grasp evaluator \(Sec\.[IV\-D](https://arxiv.org/html/2606.31200#S4.SS4)\) are analyzed to classify failures into 14 types in four groups:Pre\-contact\(3: position error, unreachable pose, approach angle\);Contact\(3: width mismatch, collision, orientation\);Post\-grasp\(4: slip, drop, force damage, deformation\);Task\-level\(4: wrong object, constraint violation, timeout, unknown\)\. Each type maps to a correction rule \(e\.g\.,Slip→\\rightarrowf′=1\.2​ff^\{\\prime\}\{=\}1\.2f;Width\_Mismatch→\\rightarroww′=w\+0\.01w^\{\\prime\}\{=\}w\{\+\}0\.01m\)\.

Corrections escalate through three levels \(Fig\.[4](https://arxiv.org/html/2606.31200#S4.F4)\): Level 1 \(r≤1r\{\\leq\}1\): targeted parameter tuning \(force, width, height\); Level 2 \(r=2r\{=\}2\): grasp type rotation \(power→\\rightarrowpinch→\\rightarrowside\) with 10% force reduction; Level 3 \(r=3r\{=\}3\): complete re\-planning with reset parameters and random perturbations \(ϵ∼𝒰​\(−0\.01,0\.01\)3\\boldsymbol\{\\epsilon\}\\sim\\mathcal\{U\}\(\-0\.01,0\.01\)^\{3\}\)\.

### IV\-CEpisodic Memory and Cross\-Task Transfer

Successful grasps are stored in session\-level episodic memoryℳ=\{\(cj,𝐠j∗\)\}\\mathcal\{M\}=\\\{\(c\_\{j\},\\mathbf\{g\}\_\{j\}^\{\*\}\)\\\}indexed by object category, enabling within\-session transfer: proven parameters for one cup transfer directly to subsequent cups in table\-clearing tasks, eliminating exploration overhead\.

![Refer to caption](https://arxiv.org/html/2606.31200v1/figures/fig.4.png)Figure 4:Agentic self\-reflective planning loop\. The ReAct cycle generates Thought→\\toAction→\\toObservation\. Failures trigger reflection using a 14\-type taxonomy, escalating through three retry levels\. Successes are stored in episodic memoryℳ\\mathcal\{M\}for cross\-task transfer\.
### IV\-DGrasp Quality Model

Central to the failure diagnosis and recovery described above, the analytical grasp quality model evaluates each grasp attempt using a seven\-factor scoring function that captures complementary aspects of grasp feasibility and robustness:

Q​\(𝐠,o\)=∑k=17ωk⋅ϕk​\(𝐠,o\)Q\(\\mathbf\{g\},o\)=\\sum\_\{k=1\}^\{7\}\\omega\_\{k\}\\cdot\\phi\_\{k\}\(\\mathbf\{g\},o\)\(2\)where the seven factors cover position accuracy \(ω1=0\.20\\omega\_\{1\}\{=\}0\.20\), width compatibility \(ω2=0\.15\\omega\_\{2\}\{=\}0\.15\), force appropriateness \(ω3=0\.10\\omega\_\{3\}\{=\}0\.10\), grasp type matching \(ω4=0\.15\\omega\_\{4\}\{=\}0\.15\), approach clearance \(ω5=0\.15\\omega\_\{5\}\{=\}0\.15\), object difficulty \(ω6=0\.10\\omega\_\{6\}\{=\}0\.10\), and grip security \(ω7=0\.15\\omega\_\{7\}\{=\}0\.15\)\. Failure modes are triggered when anyϕk\\phi\_\{k\}falls below its thresholdθk\\theta\_\{k\}\(e\.g\.,ϕ2<0\.4⇒\\phi\_\{2\}<0\.4\\RightarrowWidth\_Mismatch\)\. This ensures*physically grounded*and*reproducible*outcomes\.

## VExperimental Analysis

### V\-AExperimental Setup

Simulation Environment\.We conduct experiments in an analytical simulation modeling a Franka Emika Panda 7\-DoF arm with a parallel\-jaw gripper \(0–80 mm aperture\) on a0\.6×0\.80\.6\\times 0\.8m tabletop with household objects of diverse shapes, materials, and fragility \(Table[I](https://arxiv.org/html/2606.31200#S5.T1)\)\. The simulator implements forward/inverse kinematics with joint limits, geometric collision detection using a gripper envelope model, and a seven\-factor grasp quality model \(Eq\.[2](https://arxiv.org/html/2606.31200#S4.E2)\) for reproducible outcome determination\.

Hardware and Model Configuration\.We use Qwen3\-VL\-8B\[[23](https://arxiv.org/html/2606.31200#bib.bib23)\]as the foundation VLM on a single NVIDIA RTX 5090 GPU \(32 GB\)\. INT4 quantization via BitsAndBytes\[[24](https://arxiv.org/html/2606.31200#bib.bib24)\]with FP8 KV cache achieves 155\.5 tokens/s \(1\.78×\\timesover FP16\)\. A complete trial takes∼\\sim21 s \(SG\), 25 s \(IT\), and 37 s \(LH\)\.

Object Dataset\.We curate 12 household objects spanning 10 categories with diverse physical properties \(Table[I](https://arxiv.org/html/2606.31200#S5.T1)\), covering 7 of the 8 affordance types with fragility ranging from 0\.0 \(wood cube\) to 0\.9 \(wine glass\)\.

TABLE I:Object Dataset: 12 objects spanning 10 categories and 8 affordance types\. Fragility ranges from 0\.0 \(durable\) to 0\.9 \(very fragile\)\.Task Suite\.Following the multi\-category evaluation protocol of\[[16](https://arxiv.org/html/2606.31200#bib.bib16)\], we design 12 benchmark tasks organized in three categories of increasing complexity:

- •Single\-Grasp \(SG, 6 tasks\):Pick up an individual object from an uncluttered tabletop: cup, apple, bottle, banana, bowl, cube\. Difficulty ranges from easy \(banana: elongated, wrappable\) to hard \(bowl: edge grasp, narrow rim exceeding gripper width\)\.
- •Interactive \(IT, 4 tasks\):Grasp with spatial constraints: \(a\) cup near a fragile glass vase \(requires cautious approach\), \(b\) ball behind a box \(occlusion\-aware planning\), \(c\) phone near a wine glass \(fragility with flat object geometry\), \(d\) screwdriver requiring tool\-appropriate side grasp\. These tasks evaluate scene graph reasoning and constraint satisfaction\.
- •Long\-Horizon \(LH, 2 tasks\):Sequentially clear 3 objects, requiring episodic memory to transfer successful parameters across sequential attempts: \(a\) clear a table with cup, apple, and cube, \(b\) sort 3 objects by fragility level\.

Each task receives 30 independent trials \(360 total per configuration across all 12 tasks\)\.

Evaluation Protocol\.We evaluate through comprehensive ablation under identical conditions using the analytical quality model \(Eq\.[2](https://arxiv.org/html/2606.31200#S4.E2)\)\. The primary baseline isVLM\-Only\(Qwen3\-VL\-8B without RAG, scene graph, recovery, or memory\)\. For context, published methods achieve 88–97% on standard single\-object benchmarks \(GR\-ConvNet\[[1](https://arxiv.org/html/2606.31200#bib.bib1)\]: 97\.7%, AnyGrasp\[[2](https://arxiv.org/html/2606.31200#bib.bib2)\]: 88%\), but these do not include interactive or long\-horizon tasks, precluding direct comparison\. Statistical testing uses 5 complete repetitions with different random seeds\.

### V\-BMain Results: Component Analysis

TABLE II:Component Analysison 12\-task benchmark\. Success rate \(%\) per task category and overall \(30 trials each\)\.Δ\\Delta: change vs\. Full System\. All evaluated identically\. Best inbold\.Table[II](https://arxiv.org/html/2606.31200#S5.T2)presents the comprehensive component analysis\. Agentic RAG\-VLM achieves78\.3% overall success rate\(78\.7±1\.8%78\.7\\pm 1\.8\\%over 5 runs, 95% CI: \[76\.4, 81\.0\]\)\.Single\-grasp tasksachieve 91\.7%, with simple objects \(banana, cube\) reaching 100% and more challenging objects \(apple: 60\.0%, cup: 90\.0%\) requiring affordance\-specific grasping\.Interactive tasks\(64\.2%\) demonstrate the critical benefit of scene graph reasoning for constraint\-aware manipulation near fragile objects\.Long\-horizon tasks\(66\.7%\) remain challenging, as sequential dependencies amplify individual failure probabilities across multi\-step executions\. Fig\.[5](https://arxiv.org/html/2606.31200#S5.F5)illustrates representative execution traces for each category\.

A clear hierarchy of component importance emerges:HAA\-RAG is indispensable\(Δ=−53\.3%\\Delta=\-53\.3\\%\)—removing retrieval produces the largest performance collapse, with IT and LH dropping to 0%\. This confirms that affordance\-matched retrieval forms the foundation of effective grasping: without it, default parameters fail every interactive and long\-horizon task, as these require object\-specific grasp configurations\.Adaptive recovery\(Δ=−31\.6%\\Delta=\-31\.6\\%\) is the second most critical component, with IT task success dropping from 64\.2% to 33\.3% and LH from 66\.7% to 23\.3%—both task categories require iterative correction\.Scene graph reasoning\(Δ=−13\.0%\\Delta=\-13\.0\\%\) has major impact on interactive tasks \(IT: 64\.2%→\\to25\.0%,ΔIT=−39\.2%\\Delta\_\{\\text\{IT\}\}=\-39\.2\\%\), providing collision\-aware spatial planning that enables safe manipulation near fragile objects—without it, 3 of 4 interactive tasks fail entirely due to gripper–neighbor collisions\.Episodic memoryprimarily impacts long\-horizon tasks \(LH: 66\.7%→\\to55\.0%,ΔLH=−11\.7%\\Delta\_\{\\text\{LH\}\}=\-11\.7\\%\), confirming that within\-trial parameter transfer benefits sequential multi\-object tasks\.

TheHeuristic baseline\(size/fragility\-based parameter selection with recovery, no VLM or RAG\) achieves 74\.7% overall, excelling on single\-grasp \(100%\) and long\-horizon \(98\.3%\) tasks where object\-adapted parameters suffice\. However, it drops to 25\.0% on interactive tasks—matching w/o Scene Graph—because it lacks spatial reasoning to avoid collisions near fragile neighbors\. This confirms that the full system’s interactive advantage derives specifically from scene graph constraint inference, not from general parameter optimization\. Notably, w/o HAA\-RAG converges to identical performance as VLM\-Only \(25\.0%\), demonstrating that downstream modules are*ineffective*without a retrieval knowledge foundation\. The most frequent failures areWidth\_Mismatch\(gripper\-object incompatibility\),Slip\(insufficient grip force\), andForce\_Damage\(excessive force on fragile objects\), reflecting fundamental parallel\-jaw gripper limitations—the bowl diameter \(12 cm\) exceeding gripper aperture \(8 cm\) accounts for the majority of single\-grasp failures\.

![Refer to caption](https://arxiv.org/html/2606.31200v1/figures/fig.5.png)Figure 5:Qualitative execution traces across the three task categories\.SG: direct four\-step pipeline succeeding on the first attempt\.IT: constraint\-aware manipulation with slip failure→\\toL1 retry→\\torecovery\.LH: three\-object clearing task with episodic memory transfer accelerating the second same\-category grasp\.
### V\-CRecovery Mechanism Effectiveness

TABLE III:Recovery Mechanism Effectivenessacross task categories\. 30 trials per task, average over category\.To further evaluate the recovery mechanism in isolation, we compare the full Agentic RAG\-VLM against a single\-attempt variant \(Table[III](https://arxiv.org/html/2606.31200#S5.T3)\)\. For single\-grasp tasks, recovery provides\+28\.4%\+28\.4\\%improvement, primarily correcting minor force and width mismatches through Level 1 parameter tuning at a modest time overhead \(\+0\.6\+0\.6s average\)\. Interactive tasks show a\+30\.9%\+30\.9\\%gain: first\-attempt failures \(e\.g\., collision near a fragile neighbor\) are systematically diagnosed and corrected with constraint\-aware adjustments across multiple retry levels\. Long\-horizon tasks improve by\+43\.4%\+43\.4\\%because the recovery mechanism corrects individual sub\-goal failures within multi\-step sequences, preventing early failure cascading\. The time overhead scales with task complexity:\+0\.6\+0\.6s for SG,\+1\.0\+1\.0s for IT,\+3\.3\+3\.3s for LH—reflecting that harder tasks require more retry iterations \(average 2\.48 attempts for the full system, with 53\.3% recovery rate\)\.

### V\-DRetrieval Quality Analysis

TABLE IV:Retrieval Quality Evaluationon 12 object queries against a 116\-entry knowledge base\.Table[IV](https://arxiv.org/html/2606.31200#S5.T4)compares retrieval quality against aCLIP\-Onlybaseline \(category filtering \+ visual re\-ranking, skipping affordance matching\)\. HAA\-RAG achieves 91\.7% P@1 and 0\.917 MRR vs\. 66\.7% and 0\.729 for CLIP\-Only\. The\+25\.0\+25\.0pp P@1 improvement confirms that affordance\-aware filtering is critical: visual similarity alone frequently selects geometrically similar objects requiring different grasp strategies\. The affordance match rate gap \(91\.7% vs\. 63\.3%\) shows CLIP\-Only fails to select physically appropriate strategies for over one\-third of queries\.

### V\-EScene Graph Constraint Analysis

TABLE V:Scene Graph Reasoning Impacton interactive tasks \(120 trials total\)\.Table[V](https://arxiv.org/html/2606.31200#S5.T5)isolates the scene graph’s impact on interactive tasks\. Without it, success drops to 25\.0% due to gripper–neighbor collisions\. Scene graph construction yields\+33\.3\+33\.3pp improvement \(25\.0%→\\to58\.3%\), and adding constraint inference enables 75% constraint satisfaction, covering content preservation, collision avoidance, and force reduction\.

### V\-FOut\-of\-Distribution Generalization

To evaluate generalization, we test on In\-Distribution \(KB objects\), Near\-OOD \(same categories, different instances\), and Far\-OOD \(unseen categories\)\. The system degrades from 84\.4% \(In\-Dist\) to 54\.4% \(Far\-OOD\), expected since Far\-OOD objects lack directly matching affordance entries\. Nevertheless, HAA\-RAG’s affordance\-based retrieval enables meaningful cross\-category transfer \(e\.g\., bottle\-body\-grasp for an unseen thermos\), and the retry mechanism maintains above 54% success on entirely unseen categories\.

## VIConclusion

This paper presents Agentic RAG\-VLM, a unified framework for robotic grasping that integrates affordance\-aware retrieval\-augmented generation with scene graph constraint reasoning and agentic self\-reflective planning to bridge the gap between VLM\-based semantic understanding and physically grounded grasp execution\. By unifying hierarchical affordance retrieval with spatial reasoning and closed\-loop failure recovery, the system demonstrates robust adaptability across diverse scenarios, from single\-object manipulation to complex multi\-stage table\-clearing operations\.

Key innovations include HAA\-RAG, a three\-level retrieval pipeline that matches grasp strategies by functional affordance rather than visual similarity, forming the indispensable foundation of effective grasping; a Scene Graph Constraint Reasoner that translates spatial relationships into concrete grasp parameter adjustments for safe manipulation near fragile neighbors; and a structured failure diagnosis mechanism with 14\-type taxonomy and three\-level adaptive retry that provides physics\-grounded closed\-loop refinement, particularly critical for long\-horizon tasks\. By decoupling affordance\-level reasoning from low\-level grasp execution through a modular knowledge\-driven architecture, Agentic RAG\-VLM enables generalizable manipulation without object\-specific training\.

Future work will focus on scaling the knowledge base through autonomous experience acquisition during deployment, extending affordance reasoning to dexterous multi\-finger hands\[[16](https://arxiv.org/html/2606.31200#bib.bib16)\]with richer contact geometries, and enhancing cross\-category generalization using hierarchical affordance ontologies with causal reasoning\. This research represents a step toward general\-purpose manipulation systems capable of operating reliably across diverse objects and environments with minimal reconfiguration\.

## References

- \[1\]S\. Kumra, S\. Joshi, and F\. Sahin, “Antipodal robotic grasping using generative residual convolutional neural network,” in*IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\)*, 2020\.
- \[2\]H\.\-S\. Fang, C\. Wang, H\. Fang, M\. Gou, J\. Liu, H\. Yan, W\. Liu, Y\. Xie, and C\. Lu, “AnyGrasp: Robust and efficient grasp perception in spatial and temporal domains,”*IEEE Transactions on Robotics \(T\-RO\)*, 2023\.
- \[3\]M\. Shridhar, L\. Manuelli, and D\. Fox, “CLIPort: What and where pathways for robotic manipulation,” in*Conference on Robot Learning \(CoRL\)*, 2022\.
- \[4\]A\. Brohan, N\. Brown, J\. Carbajal, Y\. Chebotar, X\. Chen, K\. Choromanski*et al\.*, “RT\-2: Vision\-language\-action models transfer web knowledge to robotic control,”*arXiv preprint arXiv:2307\.15818*, 2023\.
- \[5\]M\. Ahn, A\. Brohan, N\. Brown, Y\. Chebotar, O\. Cortes*et al\.*, “Do as i can, not as i say: Grounding language in robotic affordances,” in*Conference on Robot Learning \(CoRL\)*, 2022\.
- \[6\]W\. Huang, C\. Wang, R\. Zhang, Y\. Li, J\. Wu, and L\. Fei\-Fei, “VoxPoser: Composable 3d value maps for robotic manipulation with language models,” in*Conference on Robot Learning \(CoRL\)*, 2023\.
- \[7\]J\. Liang, W\. Huang, F\. Xia, P\. Xu, K\. Hausman, B\. Ichter, P\. Florence, and A\. Zeng, “Code as policies: Language model programs for embodied control,” in*IEEE International Conference on Robotics and Automation \(ICRA\)*, 2023\.
- \[8\]X\. Li, M\. Zhang, Y\. Geng, H\. Geng, Y\. Long, Y\. Shen, H\. Wang*et al\.*, “ManipLLM: Embodied multimodal large language model for object\-centric robotic manipulation,” in*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024\.
- \[9\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh*et al\.*, “Learning transferable visual models from natural language supervision,” in*International Conference on Machine Learning \(ICML\)*, 2021\.
- \[10\]J\. J\. Gibson, “The theory of affordances,”*Hilldale, USA*, vol\. 1, no\. 2, pp\. 67–82, 1977\.
- \[11\]A\. Zeng, P\. Florence, J\. Tompson, S\. Welker, J\. Chien, M\. Attarian*et al\.*, “Transporter networks: Rearranging the visual world for robotic manipulation,” in*Conference on Robot Learning \(CoRL\)*, 2020\.
- \[12\]B\. Chen, Z\. Xu, S\. Kirmani, B\. Ichter, D\. Driess, P\. Florence, D\. Sadigh, L\. Guibas, and F\. Xia, “SpatialVLM: Endowing vision\-language models with spatial reasoning capabilities,”*arXiv preprint arXiv:2401\.12168*, 2024\.
- \[13\]M\. Sundermeyer, A\. Mousavian, R\. Triebel, and D\. Fox, “Contact\-GraspNet: Efficient 6\-dof grasp generation in cluttered scenes,” in*IEEE International Conference on Robotics and Automation \(ICRA\)*, 2021\.
- \[14\]W\. Huang, F\. Xia, T\. Xiao, H\. Chan, J\. Liang, P\. Florence, A\. Zeng*et al\.*, “Inner monologue: Embodied reasoning through planning with language models,” in*Conference on Robot Learning \(CoRL\)*, 2022\.
- \[15\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- \[16\]H\. Liu*et al\.*, “RoboDexVLM: Visual language model\-enabled task planning and motion control for dexterous robot manipulation,”*arXiv preprint arXiv:2503\.01616*, 2025\.
- \[17\]N\. Wake, A\. Kanehira, K\. Sasabuchi, J\. Takamatsu, and K\. Ikeuchi, “GPT\-4V\(ision\) for robotics: Multimodal task planning from human demonstration,”*IEEE Robotics and Automation Letters \(RA\-L\)*, 2024\.
- \[18\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal*et al\.*, “Retrieval\-augmented generation for knowledge\-intensive NLP tasks,” in*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- \[19\]Z\. Liu, A\. Bahety, and S\. Song, “REFLECT: Summarizing robot experiences for failure explanation and correction,” in*Conference on Robot Learning \(CoRL\)*, 2023\.
- \[20\]P\. Mandikal and K\. Grauman, “DexVIP: Learning dexterous grasping with human hand pose priors from video,”*arXiv preprint arXiv:2202\.00164*, 2022\.
- \[21\]K\. Mo, L\. J\. Guibas, M\. Mukadam, A\. Gupta, and S\. Mittal, “Where2Act: From pixels to actions for articulated 3d objects,” in*IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2021\.
- \[22\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao, “ReAct: Synergizing reasoning and acting in language models,” in*International Conference on Learning Representations \(ICLR\)*, 2023\.
- \[23\]S\. Bai, J\. Bai, A\. Yang, P\. Wang, J\. Lin, and C\. Zhou, “Qwen3\-VL technical report,”*arXiv preprint arXiv:2511\.21631*, 2025\.
- \[24\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer, “QLoRA: Efficient finetuning of quantized language models,” in*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.