ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop

# An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop
Source: [https://arxiv.org/html/2607.24770](https://arxiv.org/html/2607.24770)
,Subrata BiswasWorcester Polytechnic InstituteWorcesterUSA[sbiswas@wpi\.edu](https://arxiv.org/html/2607.24770v1/mailto:[email protected]),Bashima IslamWorcester Polytechnic InstituteWorcesterUSA[bislam@wpi\.edu](https://arxiv.org/html/2607.24770v1/mailto:[email protected])andSai Swaminathan[sswamin6@utk\.edu](https://arxiv.org/html/2607.24770v1/mailto:[email protected])University of Tennessee KnoxvilleKnoxvilleUSA

###### Abstract\.

Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions\. Prior multimodal assistants have shown promise for procedural guidance, but most rely on cloud inference and fixed always\-on perception, making them poorly suited to privacy\-sensitive, latency\-critical domestic settings\. We presentProcAgent, a fully on\-device, agentic, vision\-based procedural assistant for real\-time adaptive guidances on a single NVIDIA Jetson AGX Orin\.ProcAgentuses a propose\-and\-verify architecture that combines low\-latency continuous perception, a symbolic task graph, on\-demand vision\-language verification, and an LLM\-based interaction agent\. The system continuously proposes user progress, invokes expensive visual reasoning only when ambiguity or likely deviation arises, and supports both reactive question answering and proactive intervention with human\-in\-the\-loop confirmation\. We evaluateProcAgentalong four dimensions: perception accuracy, reasoning, task\-level performance, and user experience\. Despite running entirely on\-device, the system maintains responsive interaction, resolving text\-only queries in approximately 2 seconds and visually grounded queries in approximately 8 seconds\. In a user study with 10 participants completing assembly tasks,ProcAgentreceives positive ratings for comprehensibility, actionability, and privacy comfort\. These results show that adaptive procedural assistance can be achieved entirely on edge hardware without sacrificing usability\.

††ccs:Human\-centered computing††ccs:Human\-centered computing Ubiquitous and mobile computing††ccs:Human\-centered computing Ubiquitous and mobile computing systems and tools## 1\.Introduction

![Refer to caption](https://arxiv.org/html/2607.24770v1/x1.png)Figure 1\.ProcAgent, deployed on home assitant device, supports furniture assembly in both reactive and proactive modes\. In thereactive mode \(A\-C\), the user explicitly requests help, asks for the next assembly step, and receives visual verification of leg alignment\. In theproactive mode\(D\-E\),ProcAgentmonitors the task and intervenes when error occurs to confirm whether the legs are securely attached and whether the table has been flipped before lifting, thereby preventing procedural mistakes\. \(F\) shows theProcAgent’s deployment on home assistant device with it’s hardware, including a camera, speaker and microphone, and an NVIDIA Jetson AGX Orin module for perception, interaction, and computation\.Procedural tasks, such as furniture assembly, home repair, laboratory protocols, craft fabrication, share a simple structure: a sequence of discrete physical actions that transform components into a finished artifact\. Yet executing them imposes substantial cognitive demand\. The user must simultaneously parse instructions, maintain a mental model of current state, locate parts, plan the next action, and monitor for errors, all while performing the physical manipulation itself\(Eeseeet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib46)\)\. This concurrent load on working memory, spatial reasoning, and attention is especially taxing for non\-experts\(Sewellet al\.,[2016](https://arxiv.org/html/2607.24770#bib.bib47)\)\. This demand is not uniform across the task\. A user executing a straightforward step needs no assistance; a user stuck on an ambiguous diagram at step 17 of 38 needs immediate, spatially grounded guidance\. Static manuals cannot make this distinction, and AR overlays or pre\-recorded videos, though richer, remain non\-adaptive: they cannot detect errors, recover from deviations, or answer free\-form questions grounded in the current workspace state\(Hartantoet al\.,[2019](https://arxiv.org/html/2607.24770#bib.bib48)\)\. What is needed is a just\-in\-time assistant that monitors progress, detects confusion, and intervenes only when warranted\.

A growing body of work has begun to address this need\. Recent datasets such as ProMQA\-Assembly\(Hasegawaet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib50)\)and CaptainCook4D\(Peddiet al\.,[2024](https://arxiv.org/html/2607.24770#bib.bib49)\)have established procedural QA and error\-detection benchmarks, while systems like PrISM\-Q&A\(Arakawaet al\.,[2024](https://arxiv.org/html/2607.24770#bib.bib45)\)demonstrate step\-aware voice assistants that pair multimodal procedure tracking with LLMs, and CHEF\-VL\(Wanget al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib30)\)shows that vision\-language models can detect cognitive sequencing errors in cooking\. Other efforts ground task assistance in single demonstrations or push real\-time action recognition onto embedded platforms\(Menget al\.,[2008](https://arxiv.org/html/2607.24770#bib.bib51); Luoet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib52)\)\. Collectively, these results validate that LLM\- and VLM\-driven procedural assistants are both feasible and useful\. Yet two limitations cut across this prior work\. First, most systems rely on cloud inference for the LLM or VLM components\(Menget al\.,[2008](https://arxiv.org/html/2607.24770#bib.bib51)\), which is problematic for a vision\-based assistant that must continuously observe the user’s workspace\. Streaming this video to remote servers raises well\-documented privacy and “surveillance anxiety” concerns in domestic settings\(Dourish,[2004](https://arxiv.org/html/2607.24770#bib.bib53)\), introduces 5–10 second round\-trip latencies that break the just\-in\-time coaching loop, and makes the system unusable in network\-constrained environments\. Second, procedural tasks are inherently multimodal: some turns require only language reasoning, others visual grounding, others knowledge retrieval, and many require all three\. Existing systems treat perception as either always\-on, incurring continuous compute and thermal cost, or as a fixed pipeline applied uniformly to every turn regardless of complexity\. Neither scales to sustained real\-time interaction\.

What is missing is an*agentic*architecture that dynamically selects the appropriate tool per turn – invoking vision only when spatial verification is needed, retrieving knowledge only when procedural detail is required, and otherwise relying on language reasoning alone, and that does so entirely on\-device\. Achieving this on commodity edge hardware \(e\.g\., NVIDIA Jetson\-class devices\) is non\-trivial: continuous billion\-parameter VLM inference exhausts unified memory and triggers thermal throttling within minutes, while lightweight detectors lack the semantic granularity to reliably distinguish fine\-grained procedural states such as a partially aligned leg from a fully secured one\(Huanget al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib58)\)\. The architectural question, then, is not whether LLM agents*can*assist with procedural tasks, prior work has shown they can, but how to deliver agentic intelligence under the joint constraints of privacy, latency, and edge compute\.

We address these constraints withProcAgent, a fully on\-device, agentic, vision\-based procedural assistant for real\-time adaptive coaching in domestic environments\.ProcAgentruns entirely on a single NVIDIA Jetson AGX Orin and is built around a*propose\-and\-verify*paradigm\. A lightweight perception module, initialized through contrastive image pretraining and fine\-tuned on a small set of task\-specific step descriptions, runs continuously and proposes what it believes the user is doing\. These proposals are treated as tentative hypotheses and do not directly update the system’s understanding of progress\. Instead, a symbolic task representation, an explicit graph of steps and their ordering rules, determines which proposals are plausible and which require confirmation\. When needed, a vision–language model is invoked on demand to verify the current state, allowing the system to use high\-fidelity visual reasoning only at critical moments rather than continuously\. An LLM\-based agent handles interaction with the user, deciding when to intervene, what to ask, and when to defer to the user’s judgment\. As illustrated in Figure 1,ProcAgentsupports both*reactive*and*proactive*interaction modes\. In reactive mode, the agent responds to user\-initiated queries, providing step guidance or verifying actions on demand\. In proactive mode, the agent continuously monitors task progress and intervenes only when it detects a likely deviation from the expected sequence, using brief confirmation to correct errors before they propagate\. Together, these components decouple*perceptual tracking*from*semantic reasoning*, allowing each to operate at the latency and fidelity appropriate to its role\. The agent further reasons about when visual grounding is necessary before invoking vision models \(e\.g\., answering “what is my next step?” from the task model while reserving vision for “is this aligned correctly?”\)\. In addition,ProcAgentincorporates a human\-in\-the\-loop confirmation strategy that defers to the user when perception and procedural structure disagree \(e\.g\., briefly asking for confirmation before correcting a suspected mistake\)\. This design reflects a key interaction principle: in voluntary\-use settings, the cost of a false intervention often exceeds that of a missed one\.

We evaluateProcAgentalong four dimensions: perception accuracy, reasoning, task\-level performance, and user experience\. On a single NVIDIA Jetson AGX Orin, the system operates entirely on\-device and supports real\-time interaction, with responses resolving in approximately 2 seconds for text\-only queries and 8 seconds for visually grounded reasoning\. Across agent backbones, we observe clear trade\-offs between response quality and tool\-use reliability, with our selected model providing the most consistent performance\. In a user study with 10 participants performing assembly tasks,ProcAgentreceives broadly positive ratings for comprehensibility, actionability, and privacy comfort, with participants explicitly valuing the system’s on\-device operation\. Together, these results demonstrate that adaptive, agentic procedural assistance can be realized entirely on edge hardware without sacrificing usability, suggesting that the trade\-off between privacy, latency, and capability is architectural rather than fundamental\. Our work makes the following contributions:

1\. An end\-to\-end, fully on\-device agentic system for procedural assistance\.

We presentProcAgent, a vision\-based procedural assistant that integrates perception, structured task tracking, and language\-based interaction into a unified system that operates entirely on edge hardware\. We demonstrate that real\-time, adaptive procedural guidance can be achieved without cloud inference while preserving usability and user trust\.

2\. A framework for making multimodal procedural assistants efficient and reliable on edge devices\.Instead of running perception and reasoning continuously, we show how lightweight, always\-on model can continuously propose the user’s actions and invoke expensive visual reasoning only when needed, while task structure and user confirmation maintain correctness under our proposed framework\.

3\. An empirical characterization of performance and user interaction in on\-device procedural assistants\.

Through system ablations and a user study, we show how different components contribute to perception accuracy, procedural correctness, resource efficiency, and user experience\. Our results identify human\-in\-the\-loop confirmation as a key driver of sequence adherence and demonstrate that on\-device deployment meaningfully improves user perceptions of privacy and trust\.

## 2\.Related Work

### 2\.1\.Vision\-Based Assistants

Early vision\-based assistants relied on fiducial markers, depth sensors, or structured lighting to track part positions and infer assembly state in controlled manufacturing environments\(Reet al\.,[2016](https://arxiv.org/html/2607.24770#bib.bib36); Radkowskiet al\.,[2015](https://arxiv.org/html/2607.24770#bib.bib37)\)\. While these systems demonstrated the feasibility of vision\-driven guidance, their dependence on instrumented environments and specialized hardware limited their applicability to real\-world, unstructured settings\.

The emergence of deep learning brought significant advances in visual recognition for procedural tasks\. Convolutional neural network\-based approaches enabled action recognition and step detection from raw video without environment instrumentation, with datasets such as the IKEA Assembly Dataset\(Ben\-Shabatet al\.,[2021](https://arxiv.org/html/2607.24770#bib.bib35)\)and Assembly101\(Seneret al\.,[2022](https://arxiv.org/html/2607.24770#bib.bib28)\)providing large\-scale benchmarks for evaluating assembly\-specific recognition\. Egocentric video understanding, explored through datasets like EPIC\-Kitchens\(Damenet al\.,[2018](https://arxiv.org/html/2607.24770#bib.bib38)\)and WEAR\(Bocket al\.,[2024](https://arxiv.org/html/2607.24770#bib.bib39)\)further demonstrated that first\-person visual perspectives could capture the fine\-grained hand\-object interactions central to procedural task performance\. These advances established that visual recognition models could reliably identify what a user was doing, but not what they should do next, nor how to communicate that to them\.

More recent systems have begun to bridge perception and guidance by pairing visual recognition with natural language generation\. AR\-based assistants such as HoloAssist\(Wanget al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib40)\)combine egocentric video understanding with step\-level instruction delivery, while systems built on large vision\-language models like InstructBLIP\(Daiet al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib33)\)and LLaVA\(Liuet al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib41)\)have shown impressive ability to answer open\-ended visual queries about procedural scenes\. In the surgical and medical domain, vision\-based guidance systems have demonstrated that real\-time visual feedback can meaningfully reduce procedural errors\(Gaoet al\.,[2021](https://arxiv.org/html/2607.24770#bib.bib32); Kletzet al\.,[2019](https://arxiv.org/html/2607.24770#bib.bib42)\)\. Despite these advances, a consistent limitation persists across the literature: visual recognition and language generation are treated as loosely coupled modules rather than components of a unified, state\-aware system\. The vision module observes, and the language module responds, but neither maintains a persistent understanding of where the user is in the task, what has already been verified, or what constraints govern what can legally happen next\.

### 2\.2\.Error Detection and Recovery Systems in Procedural Task

Detecting and recovering from errors in procedural tasks has been a longstanding challenge across robotics, human\-computer interaction, and intelligent tutoring systems\. Early approaches relied on rule\-based monitors that compared sensor readings or symbolic state representations against predefined correct sequences, flagging deviations when observed behavior diverged from the expected plan\(Hegemannet al\.,[2022](https://arxiv.org/html/2607.24770#bib.bib27)\)\. While effective in highly controlled environments, these systems were brittle in the face of real\-world variability\. Partial occlusions, user hesitation, and ambiguous intermediate states were enough to cause cascading misclassifications\.

The introduction of video\-based action recognition brought greater flexibility to error detection in procedural settings\. Systems such as Assembly101\(Seneret al\.,[2022](https://arxiv.org/html/2607.24770#bib.bib28)\)and FineGym\(Shaoet al\.,[2020](https://arxiv.org/html/2607.24770#bib.bib29)\)demonstrated that fine\-grained temporal action segmentation could reliably identify when a step was performed incorrectly or out of order\. Similarly, error detection in cooking and surgical domains\(Leaet al\.,[2017](https://arxiv.org/html/2607.24770#bib.bib31); Gaoet al\.,[2021](https://arxiv.org/html/2607.24770#bib.bib32)\)showed that temporal convolutional networks and transformer\-based architectures could capture the sequential dependencies between steps well enough to flag anomalies without explicit rule encoding\. CHEF\-VL\(Wanget al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib30)\)proposes a framework for cooking scenarios that detects sequence errors by leveraging two vision\-language models \(VLMs\): one for human activity recognition and the other for tracking state transitions\. Additionally, it introduces an action\-state merging function to reduce prediction noise\. However, these systems operate purely as passive detectors, they identify that something has gone wrong but provide no mechanism for communicating the error to the user or guiding them toward recovery\.

Additionally, LLM\-based chatbots\(Tanget al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib44)\)are conversational, either they retrieved context from knowledge base or the user interactions\. They can not be applied in multimodal agentic settings for procedural tasks where agent needs to understand the sequential progress of the user and give adaptive feedback to recover from error if user deviates\. CataractBot\(Ramjeeet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib43)\)is LLM\-based chatbot that answers patient questions, provide expert\-in\-the\-loop medical guidance, and support care\-related dialogue, but they do not maintain a step\-aware representation of an ongoing procedure\. As a result, they cannot detect whether a user has followed the correct sequence, identify illegal transitions in real time, or trigger recovery based on the current process state\.

More recent work has begun to close this gap by coupling error detection with natural language feedback\. InstructBLIP\(Daiet al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib33)\)and similar vision\-language systems have been used to generate post\-hoc error explanations from visual observations, while systems like HELPER\(Sarchet al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib34)\)combine LLM\-based planning with error recovery in embodied navigation tasks\. In the procedural assembly domain, specifically, work by Ben Shabat et al\.\(Ben\-Shabatet al\.,[2021](https://arxiv.org/html/2607.24770#bib.bib35)\)on the IKEA Assembly Dataset introduced benchmarks for step recognition and order verification, laying important groundwork for systems that reason about assembly correctness\.

Nevertheless, these approaches share a common limitation: error recovery is treated as a separate, reactive process that is triggered only after a mistake has already been made and confirmed, rather than being integrated into a continuous, proactive guidance loop\. The gap our work addresses is the absence of a unified system that combines real\-time error detection, procedural constraint enforcement, and natural language recovery within a single closed\-loop agent\. Existing systems either detect errors without communicating them, communicate without grounding responses in verified visual state, or recover without preventing the same error from recurring\.

### 2\.3\.LLM\-Based Agentic Systems for Procedural Task Guidance

LLM\-based agents have expanded procedural task guidance by combining language reasoning with external tools and perception\. Early frameworks such as ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib16)\)and Toolformer\(Schicket al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib15)\)showed that language models can coordinate tools to solve multi\-step tasks, while later embodied systems extended these ideas to settings that require state tracking, action planning, and real\-time adaptation\(Ahnet al\.,[2022](https://arxiv.org/html/2607.24770#bib.bib17); Huet al\.,[2023](https://arxiv.org/html/2607.24770#bib.bib18)\)\.

For procedural assistance, prior systems have combined language and vision to support tasks such as cooking, maintenance, and manufacturing\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib19)\)\. Systems such as PrISM\-Q&A\(Arakawaet al\.,[2024](https://arxiv.org/html/2607.24770#bib.bib45)\)and 3D\-GPT\(Sunet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib21)\)can answer step\-level queries, but they largely frame guidance as retrieval and generation rather than as a closed\-loop, state\-aware process\. As a result, they provide limited support for enforcing step order, detecting invalid transitions, or recovering from user deviations\.

Structured task representations, including graph\-based models and hierarchical planners, have improved control in robotic manipulation, but these approaches are typically designed for autonomous robots in controlled environments rather than human\-facing assistance on edge devices\. MetaAgent\(Zhanget al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib24)\)introduces FSM\-based task planning through a generated multi\-agent system, but offers limited support for runtime recovery and human feedback\. AutoRT\(Ahnet al\.,[2024](https://arxiv.org/html/2607.24770#bib.bib22)\)uses a “robot constitution” to filter unsafe actions, though mainly as a pre\-execution safety mechanism rather than a continuous runtime governance layer\. Qin et al\.\(Qinet al\.,[2026](https://arxiv.org/html/2607.24770#bib.bib23)\)similarly separate monitoring and error handling from agent cognition, but do not demonstrate this design in a real\-world procedural guidance setting\.

Additionally, deployment of LLM\-based Agentic assistant on edge and resource constraint devices is much more challenging due to their high computational demands and memory requirements\(Tuliet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib25)\)\. There are some recent works on efficient LLM deployment on Edge hardwares\(Shenet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib26)\)but none of them showed how a full end\-to\-end multi\-LLMs base agentic system would deployed and seamlessly work for real\-time guidance for procedural tasks\. SELA\(Tuliet al\.,[2025](https://arxiv.org/html/2607.24770#bib.bib25)\)uses ensembling model technique for hosting only a lightweight complexity and time critically score predictors with early exit constraint to select the appropriate LLM hosted on cloud\. It dooes not provide any framework to run everything on device in offline manner\.

Overall, prior work leaves three important gaps: deciding when visual verification is actually needed, enforcing procedural state transitions within a conversational loop, and incorporating human feedback as a primary signal during execution\.ProcAgentaddresses these gaps through selective perception, FSM\-grounded state enforcement, and human\-in\-the\-loop confirmation, all within a fully local edge deployment\.

![Refer to caption](https://arxiv.org/html/2607.24770v1/x2.png)Figure 2\.System architecture of ProcAgent, an on\-device agentic framework for procedural assistance\.The framework combines aknowledge basecontaining RAG context and runtime task knowledge, atwo\-stage perception stackfor current\-state proposal and visual verification, afinite state task modelfor progress tracking and error detection, and a centralprocedural agentfor local, latency\-aware reasoning\. The agent integrates task knowledge, visual context, and state feedback to provide guidance to the user while processing user queries and feedback in a closed\-loop interaction\.

## 3\.ProcAgentFramework

### 3\.1\.Overview

We introduceProcAgent, an on\-device agentic framework for real\-time procedural guidance during physical assembly tasks\. We instantiate and evaluateProcAgenton IKEA furniture assembly, a domain that exercises several demands a procedural assistant must meet: a fixed but partially ordered step structure, fine\-grained physical actions that resist purely verbal description, and non\-expert users who may make ordering, orientation, or completion errors\.

Figure[2](https://arxiv.org/html/2607.24770#S2.F2)shows the main components of the framework and the data flow between them\. The full system runs locally on edge hardware, with no external cloud dependency at runtime\. Its components work together to monitor the user’s progress, validate that progress against the task structure, retrieve relevant procedural knowledge, and decide when visual grounding is needed\. At the center of the framework is the*procedural agent*, the LLM\-based component that mediates interaction with the user\. It receives user queries, decides whether a response can be generated from the current task state and retrieved procedural context, determines when visual grounding is required, produces spoken guidance, and handles follow\-up when perception and task structure disagree\. We call the agent’s running record of progress, errors, and recent interaction the Session Context and describe it in §[3\.5](https://arxiv.org/html/2607.24770#S3.SS5)\.

The*knowledge base*\(K in Figure[2](https://arxiv.org/html/2607.24770#S2.F2)\) provides the procedural context used by the agent and task model\. It is constructed offline from task demonstrations and stores both RAG context for user\-facing guidance and the structured task representation used to instantiate the runtime finite\-state task model\. We describe both the construction and the structure of the knowledge base in §[4\.2](https://arxiv.org/html/2607.24770#S4.SS2)\.

The*two\-stage perception stack*\(V in Figure[2](https://arxiv.org/html/2607.24770#S2.F2)\) watches the workspace\. A low\-latency proposer runs continuously and emits tentative current\-state proposals; a higher\-capacity visual verifier is invoked selectively to confirm proposed state transitions or provide query\-grounded visual context\. This separation letsProcAgentmaintain continuous awareness without running high\-cost vision\-language reasoning on every frame\. We describe both stages in §[3\.4](https://arxiv.org/html/2607.24770#S3.SS4)\.

The*finite\-state task model*\(F in Figure[2](https://arxiv.org/html/2607.24770#S2.F2)\) tracks progress and validates proposals against the legal structure of the task\. Verified and legal events update the runtime FSM state and produce a current\-progress report for the procedural agent; verified but illegal events are escalated to the agent for follow\-up with the user\. We describe its structure in §[3\.3](https://arxiv.org/html/2607.24770#S3.SS3)and its offline construction in §[4\.2](https://arxiv.org/html/2607.24770#S4.SS2)\.

Together, these components implement the propose\-and\-verify loop traced by the arrows in Figure[2](https://arxiv.org/html/2607.24770#S2.F2)\. The proposer surfaces candidate actions, the task model filters them against the current procedural state, and the visual verifier confirms only meaningful possible transitions\. In parallel, user queries flow directly to the procedural agent: queries answerable from current progress and RAG context are handled without vision, while spatially grounded queries are routed to the verifier for query\-grounded visual context\. This architecture reflects the central design principle ofProcAgent: expensive visual reasoning should be invoked only when it is needed for state verification or user\-facing visual grounding\. The remainder of this section walks through a representative session \(§[3\.2](https://arxiv.org/html/2607.24770#S3.SS2)\) and then details each component in turn \(§[3\.3](https://arxiv.org/html/2607.24770#S3.SS3)–§[3\.5](https://arxiv.org/html/2607.24770#S3.SS5)\)\.

### 3\.2\.Walkthrough

![Refer to caption](https://arxiv.org/html/2607.24770v1/x3.png)Figure 3\.Flowcharts of ProcAgent’s proactive and reactive modes\. In theproactive mode\(left\), the system continuously detects possible task\-state changes, verifies them visually, updates the finite state task model, and issues guidance when predefined intervention conditions are satisfied\. In thereactive mode\(right\), the process begins with a user query, after which the procedural agent consults the knowledge base, task model, and, when needed, visual verification to generate and confirm appropriate guidance through interactive dialogue\.We illustrate howProcAgentoperates through two interaction modes shown in Figure[3](https://arxiv.org/html/2607.24770#S3.F3): a*proactive mode*, in which the system continuously monitors task progress and intervenes only when a likely procedural deviation is detected, and a*reactive mode*, in which the user initiates a query and the agent determines whether visual grounding is needed before responding\.

##### Proactive mode\.

A user begins assembling an IKEA LACK coffee table in front of a fixed workspace camera\. At the start of the session, the procedural agent initializes the Session Context: the interaction history is empty and the finite\-state task model is in the initial phase \(Start\)\. The low\-latency proposer begins analyzing the incoming video stream\. The user picks up a leg and aligns it against the underside of the tabletop\. The proposer emits a sequence of frame\-level predictions, which the temporal\-consistency filter smooths into a single candidate event,Align and Spin\. Because this candidate indicates a possible state change, the visual verifier is invoked to confirm whether the event is actually visible in the workspace\. The verifier confirms the event\. The finite\-state task model then checks whether the event is legal under the current task state\. Because the event satisfies the task model’s guard conditions, the task state is updated\. No user\-facing message is produced and the monitoring loop continues\. This is the common case in proactive mode: visually confirmed and procedurally valid progress is committed silently\.

The same loop also handles noisy perception\. If the proposer emits a candidate event because of hand motion, occlusion, or a transient visual ambiguity, but the visual verifier does not confirm the event, the candidate is suppressed\. The task model is not updated, the procedural agent does not interrupt the user, and the system returns to monitoring\. This suppression path prevents momentary perception errors from becoming user\-facing interventions\.

A different branch occurs when the workspace evidence and task structure disagree\. Suppose the user attaches only one leg, then begins lifting the partially assembled table to reorient it\. The proposer emitsFlip Table, and the visual verifier confirms that the user appears to be flipping the table\. The finite\-state task model then evaluates the event’s guard and finds that it is not satisfied: the flip is only legal after all four legs have been attached\. Because the event is visually confirmed but procedurally invalid, the procedural agent is invoked\. Rather than immediately asserting an error or blocking progress, the agent asks the user for confirmation\. If the user confirms the deviation, the agent retrieves the relevant prerequisite guidance from the knowledge base and instructs the user to complete the missing step\. If the user rejects the system’s interpretation, the agent suppresses the intervention, preserves the prior task state, and returns to monitoring\. In this way, proactive mode treats the user as the final authority when perception and task structure disagree\.

##### Reactive mode\.

The user can also initiate interaction at any time\. In reactive mode, a user query is sent directly to the procedural agent, which reasons over the Session Context, the finite\-state task model, and retrieved procedural context from the knowledge base\. For queries that can be answered from the current task state alone, such aswhat is my next step?orhow many steps are left?, the agent responds directly without invoking the visual verifier\.

For visually grounded queries, the agent takes a different path\. If the user asksis this leg aligned correctly?oram I doing this right?, the agent determines that current workspace evidence is needed and invokes the visual verifier\. The verifier provides query\-grounded visual context, which the agent combines with the Session Context and retrieved procedural knowledge to generate guidance\. If the user indicates that the guidance is incorrect or unclear, the agent enters a brief follow\-up dialogue, using the user’s feedback to revise the response or defer to the user’s judgment\.

Together, these two modes show howProcAgentuses the same components for different interaction needs\. In proactive mode, perception initiates the loop: the proposer surfaces candidate events, the verifier confirms them, and the task model determines whether progress is legal\. In reactive mode, the user initiates the loop: the procedural agent decides whether the query can be answered semantically or requires visual grounding\. The remainder of this section develops each component in turn \(§[3\.3](https://arxiv.org/html/2607.24770#S3.SS3)–§[3\.5](https://arxiv.org/html/2607.24770#S3.SS5)\)\.

### 3\.3\.Finite\-State Task Model

The finite\-state task model encodes the procedural structure of the task\. It consists of a task graph, extracted offline from a task\-specific corpus of demonstration videos, and a runtime state tracker that maintains the user’s current position in the graph and validates candidate events proposed by the perception stack\.

#### 3\.3\.1\.Procedural Graph\.

The task graph is a counter\-augmented finite state machine, defined by four kinds of structure\. Phases partition the task into coarse stages\. Events correspond to individual sub\-steps and are grouped under a phase\. Counters are integer variables that track how many instances of each sub\-step have been completed\. Guards are conditions over the counters that determine when an event may fire\. The dynamics follow from these: an event fires when its guard is satisfied, increments one or more counters, and phase transitions are themselves events whose guards reference the counters\. So a phase advances only when its sub\-steps have completed in a configuration consistent with the next phase’s requirements\. The graph is extracted automatically from a corpus of demonstration videos; we describe the extraction procedure in[4\.2](https://arxiv.org/html/2607.24770#S4.SS2)\.

For the assembly tasks we evaluate in this paper, the task graph has four phases \(Start,Table\-Leg,Table\-Shelf,Finished\) and five events \(Pick Up Leg,Align and Spin,Flip Table,Pick Up Shelf,Attach Shelf\)\. Within the Table\-Leg phase,Pick Up LegandAlign and Spinself\-loop with guards on two counters \(ClegC\_\{\\text\{leg\}\}andCspinC\_\{\\text\{spin\}\}\) that enforce two properties: each leg must be aligned and spun after it is picked up \(soClegC\_\{\\text\{leg\}\}andCspinC\_\{\\text\{spin\}\}stay synchronized to within one\), and no more than four legs can be processed \(Cleg≤4C\_\{\\text\{leg\}\}\\leq 4,Cspin≤4C\_\{\\text\{spin\}\}\\leq 4\)\. TheFlip Tableevent transitions the phase to Table\-Shelf, with the guard that all four legs have been completed before the flip is permitted\. Figure[4](https://arxiv.org/html/2607.24770#S3.F4)shows the full task graph for this instantiation\.

The counter\-augmented structure is what lets the task graph remain compact in the presence of parallelism\. A representation that enumerated every ordering of the four legs as a separate path would produce4\!=244\!=24paths through the Table\-Leg phase alone; the counter representation collapses these into a single phase with self\-loops, at the cost of requiring the counters to be tracked explicitly at runtime\. The graph is fixed at runtime; user feedback during a session updates the active phase and the counters but does not modify the graph itself\.

![Refer to caption](https://arxiv.org/html/2607.24770v1/x4.png)Figure 4\.Counter\-augmented finite\-state task graph for the furniture assembly instantiation\. Phases \(rounded rectangles\) partition the task into coarse stages; events \(labeled arrows\) correspond to sub\-steps and update integer counters; guard conditions on events determine when each event may fire\. Self\-loops withintable\_legallow the four leg attachments to be performed in any order while still enforcing that each leg is aligned and spun after it is picked up\.
#### 3\.3\.2\.State Tracking\.

At runtime, the task state consists of the current phase and the current values of the counters associated with that phase\. The state tracker updates the task state when a candidate event proposed by the perception stack has been verified \(§[3\.4](https://arxiv.org/html/2607.24770#S3.SS4)\) and the corresponding event’s guard is satisfied\.

The task model is consulted twice per candidate\. On receipt of a candidate, it identifies the matching event out of the current phase; a candidate that matches no event is treated as a no\-op and discarded\. After the candidate has been confirmed by the visual verifier \(§[3\.4\.2](https://arxiv.org/html/2607.24770#S3.SS4.SSS2)\), the task model evaluates the event’s full guard\. A guard\-satisfying event fires: the counters are incremented and, if the event is a phase\-transition event, the phase is advanced\. A guard\-violating event is escalated to the procedural agent \(§[3\.5](https://arxiv.org/html/2607.24770#S3.SS5)\)\.

The task model also serves as a read interface when the procedural agent is answering user queries about the session\. Queries such aswhat is my next step?orhow many steps are left?are answered from the events whose guards are satisfiable under the current \(phase, counter\) state\. This query path is separate from the state\-tracking path: reading the current satisfiable events does not advance the task state\.

The task graph and the verification VLM catch different kinds of errors\. The graph rules out events whose guards are not satisfied in the current state, which captures procedurally impossible transitions such as attempting to flip the table before all four legs are attached\. The VLM rules out events whose guards would be satisfied but that the workspace does not actually show, which captures perception errors and transient misclassifications\. Neither mechanism catches the other’s failure mode: the graph does not detect a visually absent event that happens to be guard\-satisfiable, and the VLM does not detect a visually present event that happens to violate its guard\. The two checks are applied in sequence so that a committed state update has cleared both\.

### 3\.4\.Two\-Stage Perception Stack

The perception stack maintains continuous awareness of the workspace and surfaces evidence for state\-tracking decisions made by the task model\. It consists of a lightweight proposer that processes incoming video frames and emits candidate task events, and a higher\-capacity visual verifier that confirms or disconfirms each candidate before it is committed\. A key design choice inProcAgentis to treat the visual verifier as an on\-demand visual tool rather than an always\-on perception stream\. The proposer runs continuously and is cheap enough to invoke at video rate, while the visual verifier is invoked only when a candidate event requires confirmation or when the procedural agent determines that a user query depends on the current workspace\. This division lets the stack maintain low\-latency awareness without applying high\-capacity visual reasoning to every frame\.

#### 3\.4\.1\.Proposer\.

The proposer runs in the background and maps each incoming video frame to a predicted task event in a shared image\-text embedding space\. We instantiate the proposer with a CLIP model fine\-tuned on task\-specific examples \(see Section[5\.1\.3](https://arxiv.org/html/2607.24770#S5.SS1.SSS3)for the training details\)\. For each frame, the proposer computes the frame’s embedding and its nearest task\-event description under the fine\-tuned similarity function, producing a per\-frame event prediction\.

We do not treat the proposer’s per\-frame output as ground truth\. Fine\-grained procedural tasks are precisely the setting where embedding\-based classifiers are most susceptible to noise: components look similar across events, hands frequently occlude salient parts, and lighting conditions shift during a session\. Treating every frame\-level prediction as a state\-tracking signal would generate a continuous stream of false positives\. Instead, the proposer aggregates per\-frame predictions into candidate events using a sliding\-window vote\. A buffer of the most recent frame predictions is maintained, and at fixed intervals the proposer emits the majority prediction across the buffer as a single candidate\. We call this “proposer buffer\.” The window absorbs transient flickers and brief occlusions rather than propagating them to the task model\.

The proposer is indifferent to whether a given candidate represents a change of state or a continuation of the current state: it emits candidates that match the current task state as well as candidates that do not\. The task model \(§[3\.3](https://arxiv.org/html/2607.24770#S3.SS3)\) is responsible for distinguishing claimed transitions from no\-op predictions, and discards candidates that match the current state without invoking the visual verifier\. The proposer’s role is to produce stable candidates cheaply, not to reason about whether a candidate advances the task\.

#### 3\.4\.2\.Visual verifier\.

The visual verifier is a higher\-capacity vision\-language model that the stack invokes on two paths\. The first is state\-tracking verification, triggered by the task model whenever the proposer emits a candidate that corresponds to a claimed task event\. The second is query grounding, triggered by the procedural agent whenever a user query requires visual evidence from the current workspace \(§[3\.5](https://arxiv.org/html/2607.24770#S3.SS5)\)\. The two paths share the same underlying model but use different prompts and consume the model’s output differently\.

For state\-tracking verification, the prompt is generated from the specific candidate event under evaluation\. The visual verifier is shown a small set of representative frames from the proposer buffer, together with the event description, and asked whether the event is actually occurring in the workspace, returning a yes/no verification\. When the verifier returnsyes, the task model proceeds to the guard check described in §[3\.3](https://arxiv.org/html/2607.24770#S3.SS3); when it returnsno, the candidate is suppressed and the session continues without interrupting the user\. The targeted per\-event prompt is important: asking a general\-purpose visual question such aswhat is happening in the workspace?produces free\-form output that is harder to map back to a specific event and more susceptible to distraction by unrelated scene content\.

The task model also limits unnecessary verification by filtering proposer outputs before invoking the visual verifier\. Candidate events that do not correspond to a possible event from the current task state are discarded, and candidates that match the current state are treated as no\-ops\. Thus, high\-capacity verification is reserved for proposer outputs that represent potential task progress and require visual confirmation before they can be committed\.

For query grounding, the prompt is constructed by the procedural agent from the user’s query and the current Session Context, so that the model’s attention is focused on the task\-relevant properties the query is asking about\. The model’s output in this path is free\-form natural language, because the user’s question is not a yes/no verification\. The agent consumes the output as context for response generation rather than as a gating signal\. The visual verifier fires only when one of these two paths invokes it; frames without an emitted candidate or an active user query are not processed\.

### 3\.5\.Procedural Agent Design

The procedural agent is the component the user interacts with and the component that mediates between perception and the task model when user\-facing action is required\. We instantiate the agent with a pretrained large language model that, on each invocation, receives the current Session Context and a description of available actions \(retrieve from the knowledge base, invoke the verification VLM, produce a response\) and selects among them\. The agent has three responsibilities\. First, it maintains the Session Context, a structured record of the user’s progress through the current session\. Second, it mediates error handling when the task model reports that a confirmed event violates its guard, conducting a confirmation dialogue with the user and generating recovery guidance when appropriate\. Third, it handles user\-initiated queries, including the decision of whether a query can be answered from the Session Context alone or whether the verification VLM must be invoked to ground the response\. The agent does not sit in the state\-verification path for routine events: when the task model and VLM agree that an event is visually present and guard\-satisfying, the state update is committed without any involvement from the agent\.

#### 3\.5\.1\.Session Context\.

The Session Context is the agent’s working memory across a session\. It records the interaction history between user and agent, the current task state \(phase and counter values\), and a log of errors and corrections that have occurred so far\. The Session Context is updated at three points: whenever a state transition is committed by the task model, whenever the user and agent exchange an utterance, and whenever a confirmation dialogue resolves to a definite outcome\. The agent consults the Session Context before every response it produces, so that its output is conditioned on the user’s current progress rather than on a generic reading of the task\.

The choice to maintain a persistent, structured session record changes how the agent answers questions that depend on history\. A query such aswhat went wrong?has no fixed answer; the answer depends on which error the user just encountered, which step they are in, and what the agent most recently said\. The Session Context makes these dependencies explicit and addressable\. Responses that reference earlier events in the session, such as “you skipped attaching the third leg before flipping the table,” are generated from the error log and the current task state in the Session Context, not recovered from conversation history alone\.

#### 3\.5\.2\.Perception\-Loop Mediation, Error Handling and User Feedback\.

The agent’s role in the perception loop is limited to handling escalations\. Most candidates emitted by the proposer are resolved upstream: the task model discards candidates that do not claim a transition, the verification VLM suppresses those that are not visually present, and guard\-satisfying events are committed without agent involvement\. The agent is invoked only in the remaining case, when the VLM confirms a candidate but its guard is not satisfied in the current state\. The workspace evidence and the procedural structure disagree, and resolving the disagreement requires interaction with the user\.

The agent handles escalation through a confirmation dialogue rather than autonomous action\. It informs the user of the apparent deviation, grounded in the specific event whose guard was violated, and asks the user to confirm or reject the observation\. Two branches follow\. If the user confirms, the agent retrieves guidance for the prerequisite step from the knowledge base and generates a recovery instruction; the Session Context records the error and the task state is held until the user returns to a valid configuration\. If the user rejects, the agent suppresses the intervention, retains the prior task state, and does not block further progress\. Subsequent candidates are still evaluated against the task model, so a single rejected confirmation does not disable error detection for the remainder of the session\. Additionally, user can also correct the agent’s response\.ProcAgentfeatures a user\-initiated correction mechanism to resolve factual discrepancies in the agent’s output\. Upon detecting an error in the response, the user can trigger a synchronization event that shifts the agent into a structured Q&A confirmation dialogue\. In this mode, the agent systematically verifies the execution status of all prior milestones, allowing it to reconcile its internal session context and update its task graph based on validated user feedback\.

This design yields to the user on disagreement rather than blocking progress\. The rationale is practical: the VLM can misclassify, the task graph can encode a procedural expectation that the user has reasonable grounds to deviate from, and the cost of a false intervention is higher than the cost of a missed one in a voluntary\-use setting\. We discuss the tradeoffs of this choice in §[5\.4](https://arxiv.org/html/2607.24770#S5.SS4)\.

#### 3\.5\.3\.Query Handling and Response Generation\.

User queries can arrive at any point in the session and are handled on a separate path from the perception loop\. The central decision is whether a query can be answered from the Session Context and the task graph alone, or whether it requires visual grounding in the current workspace\. We refer to this decision asReason\-Before\-Perception: before any vision resources are invoked to answer a query, the agent reasons over the query text and the Session Context to decide whether visual evidence is actually necessary for the response\.

The routing decision is made by the agent’s LLM\. The LLM receives the user’s query, the current task state, and a compressed view of the Session Context, and outputs a structured decision indicating whether the verification VLM should be invoked for this query\. This is a distinct mechanism from the state\-tracking VLM invocation: in the perception loop the VLM fires on every claimed event emitted by the proposer, whereas here the VLM is invoked only when the routing decision calls for it\. Queries such aswhat is my next step?orhow many steps are left?typically route without vision, because their answers are fully determined by the current task state and the currently satisfiable events in the task graph\. Queries such asam I doing this correctly?oris this leg aligned?typically route with vision, because their answers depend on properties of the current workspace that the Session Context does not encode\.

When the routing decision calls for vision, the agent invokes the verification VLM with a prompt constructed from the query and the current Session Context, so that the model’s output addresses the step\-specific properties the query is asking about rather than the scene as a whole\. The response to the user is then generated by the agent’s LLM from three inputs: the query, the current Session Context, and context retrieved from the knowledge base\. The retrieved context consists of step\-level records for the current and adjacent task\-graph events, drawn from the same corpus of demonstration videos from which the task graph is mined \(see §[4\.2](https://arxiv.org/html/2607.24770#S4.SS2)for the construction\)\. For vision\-routed queries, the VLM’s output is added to the prompt as an additional input describing the current workspace\. This generation path is the same for both routes; the only difference is whether the VLM’s workspace description is present in the prompt\.

The Session Context that the agent provides as input to the response\-generation LLM is not the full session record\. Long interaction histories are compressed into a summary that preserves the current task state, the recent error log, and the last few user\-agent turns\. This compression is necessary because the LLM context window is bounded, and it is consistent with the routing LLM’s input, so that both the routing decision and the response generation are conditioned on the same view of the session\.

## 4\.Implementation: Task Setup and On\-Device Execution

Section[3](https://arxiv.org/html/2607.24770#S3)describedProcAgentas a task\-structured agentic framework composed of a procedural agent, a finite\-state task model, and a two\-stage perception stack\. In this section, we describe how these abstract components are instantiated in our implementation and executed on the edge device\. The implementation is organized around a separation betweentask\-specific knowledgeandtask\-independent runtime logic: the Knowledge Base and FSM configuration change across tasks, while the agent controller, perception stack, memory budgeting strategy, and inference pipeline remain fixed\.

### 4\.1\.Implementation Overview and Model Roles

To instantiateProcAgentfor a new procedural task, the developer provides a task\-specific corpus of demonstration videos and a fixed camera setup for the target workspace\.ProcAgentprocesses the demonstrations offline to extract atomic steps, validate them across models, and generate a structured Knowledge Base\. This Knowledge Base is then integrated with the generic runtime FSM engine to produce a task\-specific state model\. At deployment time, the resulting task\-specific state model is loaded into the on\-device runtime, where the lightweight proposer, visual verifier, procedural agent, and response\-generation pipeline execute locally on the edge device\.

Table[1](https://arxiv.org/html/2607.24770#S4.T1)summarizes how each framework component is instantiated in our implementation\. GPT\-OSS\-20B serves as the procedural agent for query routing, Session Context management, tool invocation, and response generation\. The two\-stage perception stack consists of a proposer implemented with fine\-tuned CLIP and a verification model implemented with Qwen2\.5\-VL\-3B\. The proposer continuously processes incoming video frames and emits low\-cost candidate task events, while the verifier is invoked only when visual grounding is required, either to confirm a proposed state transition or to answer a workspace\-dependent user query\. The finite\-state task model is implemented as a counter\-augmented FSM that enforces legal task order, validates event guards, and tracks the current phase and counter state\. The Knowledge Base is constructed offline using Gemini and GPT\-4o to extract, validate, and structure task knowledge from demonstration videos before deployment\.

Table 1\.Model roles in theProcAgentimplementation\.This mapping fixes the implementation boundary for the rest of the section: task\-specific knowledge is produced offline and loaded into the FSM and Knowledge Base, while the same on\-device proposer, visual verifier, and procedural agent are reused across task instances\.

### 4\.2\.Knowledge Base Construction and Integration

The Knowledge Base is constructed once per task instance from a corpus of demonstration videos and reused across all subsequent sessions for that instance\. Construction is performed offline using cloud language models; no cloud calls are made at inference time\. Figure[5](https://arxiv.org/html/2607.24770#S4.F5)summarizes the offline process: a three\-stage pipeline produces a structured Knowledge Base, and a separate integration step reads the Knowledge Base to instantiate the runtime finite\-state task model \(§[3\.3](https://arxiv.org/html/2607.24770#S3.SS3)\) for that instance\.

The pipeline has three stages: atomic step extraction, cross\-model validation, and hierarchical context generation\. A fourth integration step then reads the Knowledge Base produced by the pipeline and configures the runtime FSM\.

![Refer to caption](https://arxiv.org/html/2607.24770v1/x5.png)Figure 5\.Offline knowledge base construction pipeline\. Demonstration videos are processed by a three\-stage pipeline \(atomic step extraction, cross\-model validation, hierarchical context generation\) to produce the Knowledge Base\. A deterministic integration step configures the generic FSM engine for the new instance, producing the runtime\-ready finite\-state task model\.#### 4\.2\.1\.Atomic Step Extraction

The first stage uses Gemini’s long\-context video reasoning capability to perform dense temporal analysis over the demonstration videos for the instance\. The model extracts atomic steps, the smallest discrete units of action \(such as “pick up leg” or “align and spin”\), each paired with start and end timestamps\. The extraction prompt enforces a low\-level granularity heuristic so that compound or ambiguous actions are decomposed into sub\-tasks small enough to be independently verifiable by the perception stack at runtime\. The prompt is included in the Appendix[C](https://arxiv.org/html/2607.24770#A3)\.

#### 4\.2\.2\.Cross\-Model Validation

To mitigate the risk of single\-model extraction bias, a second model \(GPT\-4o\) acts as an independent validator over the Gemini\-extracted steps\. The validator audits each step for logical consistency, checks for temporal overlaps between adjacent steps, and flags any step whose description does not align with its assigned timestamp window\.

#### 4\.2\.3\.Hierarchical Context Generation

The validated steps are then organized by Gemini into a hierarchical context \(structured task description\) that the runtime system can consume\. The description has three components: a sequence of high\-level steps in their canonical execution order, a set of completion rules associating each step with a numeric completion count \(e\.g\., pick up leg: “4 times”\), and a hierarchical narrative that records inter\-step dependencies \(e\.g\., “after attaching four legs, “flip the table”\)\. Steps with repeated sub\-actions, such as “align and spin” performed once per leg, are explicitly assigned numeric repetition counts so that downstream components can ground state transitions in countable success criteria rather than soft model confidence\.

This structured task description serves two downstream uses\. The integration step reads it to configure the runtime FSM\. The procedural agent \(§[3\.5](https://arxiv.org/html/2607.24770#S3.SS5)\) retrieves from it as the RAG context that grounds response generation during live sessions: when the agent answers a user query or generates recovery guidance after an error, it draws on this description to anchor the response in the specific procedural structure of the current task instance\.

#### 4\.2\.4\.Integration with the Runtime FSM

The finite\-state task model described in §[3\.3](https://arxiv.org/html/2607.24770#S3.SS3)is implemented as a generic FSM engine, hand\-authored once and shared across all task instances\. To prepare the engine for a new instance, a deterministic integration step reads the structured task description from the Knowledge Base and configures the engine: each step in the canonical sequence becomes an event in the FSM, each completion rule becomes a counter target, and the inter\-step dependencies become guard conditions on event firing\. The phase structure \(e\.g\.,Start,Table\-Leg,Table\-Shelf,Finished\) is determined by hand\-coded rules in the integration step, applied uniformly across instances\. The output is a runtime\-ready FSM that the engine instantiates at startup\. The Knowledge Base also remains available to the procedural agent for retrieval during user queries\.

### 4\.3\.Memory Orchestration & Resource Budgeting

RunningProcAgententirely on the edge device requires coordinating multiple models under a shared memory budget\. Our implementation targets theNVIDIA Jetson AGX Orin, whose unified memory architecture exposes a single 64GB memory pool shared by the CPU and GPU\. This design simplifies data movement between components but also makes memory contention a central runtime constraint: the procedural agent, visual verifier, and proposer all draw from the same memory pool, along with their associated runtime allocations such as KV caches, vision\-encoder activations, frame buffers, audio I/O, and operating\-system overhead\. We therefore treat model placement, quantization, and tool invocation as part of the system design rather than as independent implementation details\.

We allocate memory according to the role each model plays in the agentic pipeline\. The procedural agent is the largest and most important language\-reasoning component, so GPT\-OSS\-20B is quantized to Q5\_K\_M to reduce its footprint while preserving response quality\. The visual verifier, implemented with Qwen2\.5\-VL\-3B, is also quantized to Q5\_K\_M, since it is used for visually grounded verification and query answering but is not executed continuously\. The proposer, implemented with fine\-tuned CLIP, remains small enough to run continuously for frame\-level monitoring\. This allocation leaves headroom for dynamic KV\-cache growth, vision\-encoder activations, frame buffers, audio I/O, and operating\-system overhead during interactive sessions\.

In our implementation, state\-tracking verification uses three representative frames from the proposer buffer instead of a single frame or the full window\. We sample the middle, third\-quarter, and final frames of the candidate window, giving the visual verifier short temporal context because assembly actions unfold over nearby frames and may appear as preparation, mid\-action, or completion\. The visual verifier receives these frames with the candidate event description and returns a binary decision indicating whether the event is present in the workspace\.

Before verifier invocation, the task model filters no\-op candidates and candidates that do not correspond to a possible event from the current task state\. Together, quantized model deployment, three\-frame prompting, selective verifier invocation, and task\-model filtering allowProcAgentto run a multi\-model agentic pipeline within the Jetson memory envelope\. Three\-frame prompting bounds the cost of each verification call, while task\-model filtering reduces the number of calls\. As a result,ProcAgentpreserves continuous low\-cost perception while reserving high\-cost reasoning for state verification, error handling, and visually grounded user assistance\. This avoids the sustained GPU compute pressure and memory\-bandwidth contention of an always\-on VLM stream, helpingProcAgentremain responsive under edge\-device constraints\.

### 4\.4\.Inference Pipeline

The runtime inference pipeline is implemented using*llama\.cpp*, an optimized inference runtime that supports quantized model execution on ARM\-based edge hardware\. We serve the quantized procedural agent and visual verifier through the*llama\.cpp*server interface with parameters tuned to the Jetson’s memory and compute constraints\. This allows the language and vision\-language components to run locally without cloud inference while keeping their memory footprint compatible with the shared 64GB unified memory budget\.

At the start of a session, the task\-specific FSM configuration and Knowledge Base are loaded into the runtime\. The proposer then begins processing incoming video frames continuously and emits temporally smoothed candidate events\. The inference loop handles two input streams: proposer\-generated candidates from the perception path and user\-initiated queries from the interaction path\. Candidate events are filtered by the task model and, when needed, routed to the visual verifier before any state update is committed\. User queries are routed by the procedural agent either to direct response generation from the Session Context and Knowledge Base or to visually grounded response generation using the visual verifier\.

The pipeline is structured as a non\-blocking loop so that continuous perception, visual verifier, and response generation do not unnecessarily stall one another\. The proposer continues monitoring the workspace while the agent handles user queries, and the visual verifier is invoked only for bounded verification calls or workspace\-dependent queries\. This implementation preserves the framework behavior described in Section[3](https://arxiv.org/html/2607.24770#S3)while making the runtime practical under the Jetson’s memory, compute, and latency constraints\.

## 5\.On\-Device System Evaluation

This section evaluatesProcAgentas an on\-device procedural guidance system\. We first describe the edge hardware, assembly dataset, system configurations, and evaluation metrics\. We then evaluate the low\-latency proposer, the proposer buffer and visual\-verifier frame\-selection strategy, the contribution of each system component, and the impact of procedural\-agent backbone choice\. Finally, we analyze the resource and latency trade\-offs of running the complete pipeline on the Jetson AGX Orin\.

### 5\.1\.Experimental Setup

This subsection describes the setup used for the on\-device evaluation\. We summarize the hardware, assembly dataset, system configurations, evaluation metrics, and user\-query bank used in the experiments\.

#### 5\.1\.1\.Hardware

All experiments are conducted entirely on a single NVIDIA Jetson AGX Orin\(NVIDIA,[2021](https://arxiv.org/html/2607.24770#bib.bib54)\)\(64 GB unified memory, Ampere GPU, 12\-core ARM Cortex\-A78AE\) running in MAXN power mode, with no cloud inference during evaluation\. The workspace is captured by a Logitech C920s Pro camera\(Logitech,[2026](https://arxiv.org/html/2607.24770#bib.bib55)\)\(1080p, 30,fps\); the agent’s responses are broadcast through an EMEET Luna Conference Speaker\(EMEET,[2020](https://arxiv.org/html/2607.24770#bib.bib56)\), and user queries are captured by a directional microphone\.

#### 5\.1\.2\.Evaluation Dataset

We evaluate the system on a curated subset of the IKEA Assembly Dataset\(Ben\-Shabatet al\.,[2021](https://arxiv.org/html/2607.24770#bib.bib35)\)\. From the dataset’s 371 videos, we evaluate on 15 covering three furniture types \(LACK Coffee Table, LACK Side Table, LACK TV Bench\), with five demonstrations per type\. The subset is constructed under three criteria\. First, each video must exceed 100 seconds in duration, ensuring that the demonstration includes the full assembly sequence rather than a partial or aborted attempt\. Second, we exclude videos in which the user performs multiple sub\-steps concurrently, because our task model commits to a single active step at any given time and the sequence\-adherence metric is defined with respect to that assumption; we discuss concurrent execution as future work in Section \[limitations\]\. Third, the retained videos span multiple users, lighting conditions, and table colors, to capture variation across realistic recording conditions within the dataset\. The list of selected videos is provided in Appendix[D](https://arxiv.org/html/2607.24770#A4)\.

#### 5\.1\.3\.Proposer Fine\-Tuning

The proposer \(a fine\-tuned CLIP, see §[3\.4\.1](https://arxiv.org/html/2607.24770#S3.SS4.SSS1)\) is fine\-tuned on a separate set of 29 demonstration videos drawn from the same IKEA Assembly Dataset\. The fine\-tuning videos cover the only one type of furniture but feature different users; no video used for fine\-tuning appears in the evaluation set\. No other component of the pipeline is trained on either set: the visual verifier \(Qwen2\.5\-VL\-3B\) and procedural agent \(GPT\-OSS\-20B\) are used as pretrained off\-the\-shelf models, and the Knowledge Base for each furniture type is constructed offline from different demonstration videos than the evaluation dataset\.

#### 5\.1\.4\.Baseline

We evaluateProcAgentthrough two complementary axes: an ablation across system configurations and a comparison across agent backbones\.

System Ablation\.The procedural agent and the proposer are enabled in all configurations of the system; we vary the visual verifier’s two roles, the finite\-state task model, the Knowledge Base retrieval, and the user\-confirmation dialogue\. Table[6](https://arxiv.org/html/2607.24770#S5.T6)summarizes the configurations\. Configuration A includes the procedural agent and the proposer alone; the agent generates responses from the proposer’s raw state proposals and the Session Context, with no verification, state enforcement, knowledge retrieval, or user confirmation\. Configuration B adds the visual verifier in its state\-tracking role \(§[3\.4](https://arxiv.org/html/2607.24770#S3.SS4)\)\. Configuration C adds the finite\-state task model on top of B\. Configuration D adds Knowledge Base retrieval to B without the task model, isolating retrieval’s contribution against state enforcement\. Configuration E enables both the task model and retrieval\. Configuration F adds the visual verifier in its query\-grounding role\. The fullProcAgentconfiguration adds the user\-confirmation dialogue\. The pairwise comparisons isolate the marginal contribution of each component\. A versus B measures the verifier’s effect on state tracking\. B versus C measures the task model’s effect under verifier\-only operation\. B versus D measures retrieval’s effect under verifier\-only operation\. E versus F measures the contribution of query\-grounded visual reasoning\. F versus fullProcAgentmeasures the contribution of user confirmation\.

We also evaluate a separateSingle VLMbaseline, reported in its Table[3](https://arxiv.org/html/2607.24770#S5.T3), in which a vision\-language model answers the user’s query directly from the five most recent camera frames without explicit state tracking, retrieval, verification, or user confirmation\. We treat this as an external baseline rather than an ablation configuration, since it replaces the fullProcAgentarchitecture with a one\-pass alternative\.

We additionally tested a configuration in which the proposer was disabled and the visual verifier was invoked on every incoming frame, taking on the proposer’s role of generating per\-frame state predictions\. We do not report this configuration in Table[6](https://arxiv.org/html/2607.24770#S5.T6)because per\-frame VLM invocation is not deployable in real\-time settings: a single assembly session of approximately three minutes took roughly 100 minutes to complete in this configuration, more than an order of magnitude beyond interactive latency\. The measurement is consistent with the design rationale for the propose\-and\-verify architecture \(§[3\.4](https://arxiv.org/html/2607.24770#S3.SS4)\): per\-frame verification is computationally infeasible on edge hardware, which is the gap the proposer is designed to fill\. We retain this measurement as a runtime calibration of the per\-frame vision\-language model cost rather than as an ablation row\.

For the user\-confirmation evaluation, we developed a rule\-based simulator that activates only when the system makes a sequencing error; this reflects the intended runtime role of confirmation as a corrective signal rather than a general\-purpose interaction channel\.

Agent\-Backbone Comparison\.We benchmark three quantized open\-source language models as the procedural agent: Phi\-3\-Mini\-4k\-Instruct \(3\.8B\), LLaMA\-3\.1\-Instruct \(8B\), and GPT\-OSS\-20B \(20B\)\. All three are served through llama\.cpp atQ​5​\_​K​\_​MQ5\\\_K\\\_Mquantization for LLaMA\-3\.1\-Instruct \(8B\), and GPT\-OSS\-20B \(20B\) andQ​4​\_​K​\_​MQ4\\\_K\\\_Mquantization for Mini\-4k\-Instruct \(3\.8B\) as we could not find open sourceQ​5​\_​K​\_​MQ5\\\_K\\\_Mversion of Mini\-4k\-Instruct \(3\.8B\)\. Each backbone is evaluated under the fullProcAgentconfiguration to isolate the effect of agent capacity from the rest of the pipeline\. The defaultProcAgentuses GPT\-OSS\-20B; the alternatives are evaluated for their accuracy\-latency tradeoff in §5\.2\.3\.

#### 5\.1\.5\.Metrics

We evaluateProcAgentalong three groups of metrics: decision accuracy, response latency, and response quality\. Each metric is defined operationally below\. Where a metric depends on which components a configuration includes, the relevant configuration behavior is noted in the corresponding definition\.

#### Decision Accuracy

These three metrics evaluate discrete decisions the system makes during a session, comparing each decision against ground\-truth annotations: identifying the user’s current state, classifying a candidate event as consistent or inconsistent with legal task structure, and routing a user query to the appropriate response path\.

State Accuracymeasures how often the system correctly identifies the user’s current assembly state\. It is computed at the candidate\-event level: each time the proposer emits a candidate \(after the sliding\-window vote\), we compare the system’s final predicted candidate to the ground\-truth candidate annotated for the corresponding video segment\. In configurations with the visual verifier and the finite\-state task model, the predicted candidate is the candidate that the FSM commits after verifier confirmation; in Version A, it is the candidate emitted directly by the proposer\. State Accuracy is the percentage of candidate emissions whose final predicted candidate matches the ground truth\. Ground\-truth annotations were collected from the IKEA Assembly Dataset’s\(Ben\-Shabatet al\.,[2021](https://arxiv.org/html/2607.24770#bib.bib35)\)gt\_segments label\.

Sequence Adherencemeasures how reliably the system enforces legal procedural order\. Over a session, candidate events may arise that are inconsistent with the legal task structure, for example a flip\-table event before all four legs have been attached\. The denominator is the set of candidates that reach the state\-update decision in each configuration \(proposer\-emitted in Version A, verifier\-confirmed otherwise\); the numerator is the count of those candidates that the system correctly classified as consistent or inconsistent with the legal task structure\. A score of 100% means every reaching candidate was correctly classified; a lower score reflects either an illegal candidate that was committed or a legal candidate that was incorrectly rejected\. Because the denominator differs across configurations, the metric is not a unit\-comparable quantity across rows of Table[6](https://arxiv.org/html/2607.24770#S5.T6); we report it as a per\-configuration measure of procedural\-control reliability\.

Routing F1measures the accuracy of the procedural agent’s routing decision on the user\-query path: whether a given user query is routed to direct response generation from the Session Context and Knowledge Base, or to a verifier\-grounded response \(§3\.5\.3\)\. A true positive is a routing decision that invokes the visual verifier when visual context was genuinely necessary; a false positive is a verifier invocation when language context alone was sufficient; a false negative is a query that required visual grounding but was answered without it\. Ground\-truth labels for the 20 unique query templates from the question bank \(§5\.1\.5\) were produced by a researcher’s independent annotation\. Routing F1 covers only the user\-query path\. Verifier invocation on the perception path is rule\-based: every confirmed candidate state change triggers a verifier call by construction, so this path does not involve a model decision and is not evaluated by this metric\.

#### Response Latency

Average Response Time measures the latency the user directly experiences when querying the agent, while Average Prediction Time measures the latency between a state change in the workspace and the system’s recognition of it\. Together, the two metrics characterize the responsiveness of the system on its user\-initiated and system\-initiated paths\.

Average Response Timemeasures the latency of the agent’s reactive responses to user queries, from query receipt to the start of the agent’s full grounded response\. It excludes the proactive acknowledgment \(”hold on, please”\) that the deployed system uses to signal that a longer response is being prepared; that acknowledgment is part of the proactive interaction design and is discussed in §6\. Within the metric, response time varies by query route: queries answered from the Session Context and Knowledge Base alone are bounded by the procedural agent’s generation time, while queries routed to the visual verifier additionally include the verifier’s confirmation time\.

Average Prediction Timemeasures the total time required to produce one verified candidate event, from the start of the proposer’s window to the FSM’s accept\-or\-reject decision\. This includes the proposer’s processing of its sliding window \(CLIP inference over 15 frames at stride 15\) and, when applicable, the verifier’s confirmation time\. The verifier component is zero in Version A and on candidates that the task model filters as no\-ops\. The metric is averaged over all candidate emissions in the evaluation set\.

#### Response Quality

The three response\-quality metrics distinguish three different ways an agent’s response can fail\. Factual Consistency penalizes responses that introduce information not supported by the supplied evidence \(sins of commission\)\. Correctness penalizes responses that distort or contradict the supplied evidence \(sins of distortion\)\. Helpfulness penalizes responses that, even when free of the above failures, do not move the user forward on the task \(failures of actionability\)\. The three rubrics are intended to be evaluated independently, even when they correlate empirically\.

Factual Consistency Score\(0–5\) measures whether the agent’s response is consistent with the Knowledge Base and the verified current state, in the sense that every claim in the response can be traced to one of these sources\. A score of 5 indicates a response in which all claims are supported by the supplied evidence; lower scores reflect increasing degrees of unsupported or invented content, down to 0 for a response entirely unrelated to the task\. A response that is factually true in the world but contains claims absent from the Knowledge Base is treated as inconsistent under this metric, since the metric evaluates fidelity to the supplied evidence rather than to ground truth about the world\([Kryścińskiet al\.,](https://arxiv.org/html/2607.24770#bib.bib57)\)\.

Correctness Score\(0–5\) measures whether the agent’s response correctly reflects the requirements of the current step as encoded in the Knowledge Base\. Where Factual Consistency penalizes invented claims, Correctness penalizes claims that contradict or distort the supplied evidence: for example, telling the user to attach the shelf when the Knowledge Base prescribes flipping the table at the current step\. A score of 5 indicates a response that correctly characterizes the current step and its requirements; lower scores reflect increasing degrees of factual error, down to 0 for a response that is completely wrong\.

Helpfulness Score\(0–5\) measures whether the agent’s response effectively helps the user make progress on the task\. A response can be factually consistent and correct but still unhelpful if it is vague, incomplete, or fails to specify the action the user should take next\. A score of 5 indicates a clear, specific, and immediately actionable response; lower scores reflect responses that are incomplete, unclear, or fail to move the user forward, down to 0 for a response that provides no useful guidance at all\.

The Factual Consistency, Correctness, and Helpfulness scores are produced by a panel of three large language models acting as judges \(GPT\-4o, GPT\-4\.1\-Mini, GPT\-4\.1\); the panel spans different capacities within the GPT family to verify that scores are robust to judge capability\. All three judges receive the same evaluation inputs, including the Knowledge Base supplied to the agent at session time\. The reported score is the mean across the three judges; the full prompt, per\-rubric inter\-judge agreement, and a discussion of single\-provider limitations are in Appendix[A](https://arxiv.org/html/2607.24770#A1)and §[7](https://arxiv.org/html/2607.24770#S7)\.

#### 5\.1\.6\.Question Bank

To collect realistic user queries for the ablation evaluation, we conducted a Wizard of Oz study with four participants who were not part of the user study reported in §6\. During each session, the participant assembled a LACK Coffee Table, one of the three furniture types in the evaluation set \(§5\.1\.2\)\. Responses to participant queries were delivered by a human operator from a prescripted answer set rather than by the live ProcAgent\. Decoupling query collection from the live system ensures that the queries reflect task\-driven user confusion rather than ProcAgent’s specific response behavior, making the question bank a stable evaluation set across configurations\.

We captured 53 raw utterances across the four sessions\. Each utterance corresponds to a single query addressed to the operator, segmented at speaker turns\. Unlike traditional benchmarks that filter for uniqueness, we preserved repeated and closely paraphrased queries \(e\.g\., what do I do next? uttered at multiple points in a single session\) because their recurrence at different assembly milestones is informative about how user confusion is distributed across the task\. The 53 utterances reduce to 20 unique query templates, where two utterances share a template if they differ only in surface phrasing or referenced object\.

The 20 templates were grouped into three categories\. The categories emerged from inductive coding of the utterances rather than from a target set defined in advance\. The three categories are Procedural Guidance, State Validation, and Progress and History; counts and example queries are in Table[2](https://arxiv.org/html/2607.24770#S5.T2)\. We hypothesize that the three categories map to the three response paths ProcAgent uses to answer queries: Procedural Guidance queries to the task graph alone, State Validation queries to the visual verifier, and Progress and History queries to the Session Context and Knowledge Base\. The Routing F1 metric \(§5\.1\.4\) tests this mapping by evaluating whether the procedural agent correctly routes each query template to its appropriate response path\.

Table 2\.User query categories from the Wizard of Oz study \(53 utterances across 4 participants, reducing to 20 unique templates\)\. The categories emerged inductively from the collected utterances\.

### 5\.2\.End\-to\-End System Comparison

To contextualize the benefits ofProcAgent, we compare it against aSingle VLM\(llama\-3\.1\-8b atQ​5​\_​K​\_​MQ5\\\_K\\\_Mquantization\) baseline that removes the agentic structure and relies on a single vision\-language model invocation\. Given the user’s query and the five most recent camera frames, this baseline produces a response directly, without explicit state tracking, retrieval, visual verification, or human feedback\. This provides a simple but informative point of comparison for assessing what the full architecture contributes\.

As shown in Table[3](https://arxiv.org/html/2607.24770#S5.T3),ProcAgentoutperforms the Single VLM baseline across all response\-quality measures\. Factual consistency improves from2\.54±1\.512\.54\\pm 1\.51to3\.71±1\.183\.71\\pm 1\.18, helpfulness from1\.56±1\.301\.56\\pm 1\.30to3\.43±1\.383\.43\\pm 1\.38, and correctness from1\.85±1\.401\.85\\pm 1\.40to3\.64±1\.353\.64\\pm 1\.35\. These differences suggest that assembly guidance benefits substantially from explicit procedural modeling\. Without access to verified state history, task constraints, or structured retrieval, the Single VLM can only interpret a short visual window and the current query\. As a result, its responses are often locally plausible but insufficiently grounded in the user’s actual progress through the task\.

The Single VLM baseline is faster on average \(7\.35±0\.247\.35\\pm 0\.24s versus8\.02±0\.678\.02\\pm 0\.67s forProcAgent\), which is expected given that it avoids the additional coordination and verification steps of the full system\. However, this latency advantage is modest relative to the gains in response quality\. Taken together, these results indicate that the additional structure inProcAgentintroduces limited overhead while substantially improving factual consistency, correctness, and practical usefulness\.

Table 3\.End\-to\-end comparison betweenProcAgentand a Single VLM baseline\. The Single VLM baseline responds directly from the user query and the five most recent camera frames, without explicit state tracking or other agentic components\.ProcAgentsubstantially improves factual consistency, helpfulness, and correctness, with only a modest increase in response time\. Values are mean±\\pmstandard deviation\.SystemFactual Consistency↑\\uparrowHelpfulness↑\\uparrowCorrectness↑\\uparrowAvg\. Response \(s\)↓\\downarrowSingle VLM2\.54±\\pm1\.511\.56±\\pm1\.301\.85±\\pm1\.40\\cellcolorbest7\.35±\\pm0\.24ProcAgent\\cellcolorbest3\.71±\\pm1\.18\\cellcolorbest3\.43±\\pm1\.38\\cellcolorbest3\.64±\\pm1\.358\.02±\\pm0\.67

### 5\.3\.Component Selection

In this section, we evaluate and finalize the configuration of two components: the proposer \(backbone model and temporal smoothing parameters\) and the procedural agent \(language model\)\. We select each through a comparison against alternatives\.

#### 5\.3\.1\.Proposer Backbone Selection

Table 4\.Performance comparison of various action recognition methods for furniture assembly guidance\. We report the frame\-level accuracy \(Top\-1 and Top\-3\) and Macro\-F1 score for each configuration\. Note that CLIP\-based variants are tested in few\-shot settings\.We compare CLIP against a range of supervised action\-recognition models on frame\-level classification of assembly steps\. Frame\-level Top\-1, Top\-3, and Macro\-F1 are reported for each method in Table[4](https://arxiv.org/html/2607.24770#S5.T4)\. CLIP variants are evaluated in a few\-shot setting; the supervised baselines are trained on the full task\-specific training set\. Among the supervised baselines, performance scales with model capacity\. ResNet variants reach 33\.19% to 39\.53% Top\-1 accuracy\. Temporal models perform more competitively: C3D reaches 43\.97% and P3D reaches 51\.24% Top\-1, benefiting from their ability to capture short\-range temporal dynamics\. CLIP at 64 shots surpasses all supervised baselines across every metric, achieving 65\.24% Top\-1, 89\.26% Top\-3, and 43\.31 Macro\-F1, despite operating without any task\-specific supervised training\. Beyond raw accuracy, CLIP’s few\-shot adaptability \(text descriptions plus a small number of visual examples are sufficient to accommodate new procedural steps\) and its low per\-frame inference latency make it well suited for continuous frame\-level monitoring on the Jetson AGX Orin\. We adopt CLIP at 64 shots as the proposer backbone for all subsequent experiments\.

#### 5\.3\.2\.Perception\-Stack Smoothing and Frame Selection

With CLIP fixed as the backbone, we search over the proposer’s temporal smoothing parameters and the visual verifier’s frame\-selection strategy\. We vary three parameters: the sliding\-window sizeWW, which controls how many consecutive frames must agree on a state before a candidate is emitted; the strideSS, which controls the window’s step size between emissions; and a check\-timeout threshold \(None, 15s, 20s, 25s\) that triggers a user\-confirmation event when no state change has been detected within the timeout, to confirm that the user is genuinely still in the same state\. We additionally compare three frame\-selection strategies for the verifier: Single Frame; Multi Frame \(0–100%\), which samples the first, middle, and last frames of the window; and Multi Frame \(50–100% RWS\), which samples the middle, third\-quarter, and last frames\. State Accuracy and Sequence Adherence for the full search are reported in Appendix B \(Table[8](https://arxiv.org/html/2607.24770#A2.T8)\)\.

We observe that window sizes that are too large reduce adherence by slowing candidate emission, making the system less responsive to genuine state changes\. Window sizes that are too small introduce instability: candidates are emitted too readily, leading to noisy transitions and degraded accuracy\. The configurationW=30W=30,S=15S=15,C​h​e​c​k=20​sCheck=20sstrikes the best overall balance, achieving State Accuracy of78\.25±9\.49%78\.25\\pm 9\.49\\%under the single frame condition and sequence adherence of74\.28±27\.48%74\.28\\pm 27\.48\\%under Multi Frame \(50–100% RWS\), among the highest adherence scores in the search\. WhileW=10W=10,S=5S=5,C​h​e​c​k=25Check=25s achieves a marginally higher peak accuracy, we selectW=30W=30,S=15S=15,C​h​e​c​k=20Check=20s with the Multi Frame \(50–100% RWS\) verification strategy for its superior stability and more consistent multi\-frame grounding\. This configuration is fixed for all subsequent experiments\.

Table 5\.Reasoning Agent Ablation: Impact of Central LLM Choice on System Performance\. All configurations use the fullProcAgentpipeline \(CLIP \+ Verifier \+ FSM \+ RAG\)\. Models are evaluated in quantized form on the Jetson AGX Orin\.
#### 5\.3\.3\.Agent Backbone

Table[5](https://arxiv.org/html/2607.24770#S5.T5)reports results for the three agent backbones described in §5\.1\.3 \(Phi\-3\-Mini\-4k\-Instruct, LLaMA\-3\.1\-Instruct, and GPT\-OSS\-20B\), each evaluated under the full ProcAgent configuration\. GPT\-OSS\-20B leads on every quality dimension, achieving the highest Factual Consistency \(3\.71±1\.183\.71\\pm 1\.18\), Helpfulness \(3\.43±1\.383\.43\\pm 1\.38\), Correctness \(3\.64±1\.353\.64\\pm 1\.35\), and Routing F1 \(0\.57\)\. Phi\-3\-Mini delivers the fastest response time at2\.16±0\.342\.16\\pm 0\.34seconds, but its Routing F1 of0\.260\.26indicates that it frequently fails to invoke the visual verifier at the right moment, which undermines the agent’s coordination of the rest of the pipeline\. LLaMA\-3\.1\-Instruct sits between the two on quality metrics and slightly above GPT\-OSS\-20B on response time\. The pattern is consistent across metrics: agent capacity correlates with response quality and routing accuracy, while smaller agents trade quality for latency\.

Table 6\.Architectural Ablation Study: Contribution of Each System Component\. All configurations includeProcedural Agent and Proposer; we vary remaining components\. Results are averaged over 15 assembly sessions \(IKEA LACK Coffee Table, LACK Side Table, Lack TV Bench\)\.↑\\uparrow= higher is better,↓\\downarrow= lower is better\.±\\pmdenotes one standard deviation across sessions\.The gray shaded area represents the best result and the underlines represent the second best result for each metric\.

### 5\.4\.System Ablation

The system ablation evaluates each component’s marginal contribution to procedural control and response quality\. We run the seven configurations described in §5\.1\.3 across the 15 evaluation sessions\. Because the IKEA demonstration dataset does not include user\-induced sequencing errors, user confirmation is simulated using the rule\-based mechanism described in §5\.1\.3, which activates only when the system makes a sequencing error\. Results are reported in Table[6](https://arxiv.org/html/2607.24770#S5.T6)\.

Effect of the Visual Verifier on State Tracking\.Configuration A \(the proposer alone\) achieves the highest raw State Accuracy at87\.9±9\.3%87\.9\\pm 9\.3\\%, but one of the lowest Sequence Adherence scores at42\.9±26\.7%42\.9\\pm 26\.7\\%\. The proposer often emits visually plausible candidates without enforcing procedural order\. Adding the visual verifier in Configuration B improves Sequence Adherence to47\.1±30\.0%47\.1\\pm 30\.0\\%and also lifts Factual Consistency and Correctness, indicating that visual verification contributes beyond classification accuracy by stabilizing the system’s transition decisions\.

Effect of the Finite\-State Task Model\.Adding the task model in Configuration C reduces Sequence Adherence to39\.4±32\.5%39\.4\\pm 32\.5\\%\. This may appear counterintuitive but is consistent with how the task model operates without a confirmation mechanism: the task model enforces guard conditions strictly, and when the visual signal is ambiguous, it rejects valid transitions rather than committing to uncertain state updates\. The task models’s contribution is therefore not visible in Sequence Adherence alone; it is visible in the task model’s role as a gating mechanism for verifier invocation, which produces measurable reductions in GPU load, power consumption, and thermal stress\. We report these effects in §[5\.5](https://arxiv.org/html/2607.24770#S5.SS5)\.

Effect of the Knowledge Base\.Configurations D and E \(with Knowledge Base retrieval\) improve response quality\. They achieve the highest Factual Consistency scores in the ablation:3\.89±1\.333\.89\\pm 1\.33and3\.92±1\.353\.92\\pm 1\.35respectively\. Correctness also improves to3\.57±1\.523\.57\\pm 1\.52in D and3\.63±1\.523\.63\\pm 1\.52in E\. Without retrieval, the agent recognizes the current state but lacks step\-specific procedural detail; with retrieval, responses are anchored in the requirements and completion criteria of the current step\.

Effect of the Visual Verifier on Query Grounding\.Configuration F adds the verifier on the user\-query path\. It produces the strongest Helpfulness \(3\.44±1\.323\.44\\pm 1\.32\) and Correctness \(3\.69±1\.303\.69\\pm 1\.30\) scores in the ablation\. Spatially contingent queries about part orientation, alignment, or completeness are difficult to answer from textual context and state information alone; visual grounding makes these responses concrete\. Sequence Adherence in F remains at39\.4±32\.5%39\.4\\pm 32\.5\\%, consistent with the pattern that perception and retrieval components do not by themselves enforce procedural order\.

Effect of User Confirmation\.The fullProcAgentconfiguration adds user confirmation to Configuration F\. Sequence Adherence rises from 39\.4% in F to72\.8±18\.9%72\.8\\pm 18\.9\\%inProcAgent, a 33\-point jump that is the largest single contribution in the ablation\. This identifies user confirmation as the mechanism that resolves the architectural disagreement between the visual verifier and the task model: when the verifier confirms a candidate but the task model’s guard rejects it, neither the perception evidence nor the procedural structure can adjudicate alone, and user confirmation supplies the missing signal\. The improvement comes with a modest cost: average Response Time rises to8\.02±0\.678\.02\\pm 0\.67seconds, due to the additional confirmation turn\.

Taken together, the ablation reveals an asymmetric contribution structure across metrics\. Each component improves a different aspect of system behavior: the visual verifier improves response correctness; the Knowledge Base improves factual grounding; the task model provides procedural structure and resource efficiency \(§5\.4\); the visual verifier on the query path produces the strongest spatial grounding for user questions; and user confirmation drives Sequence Adherence by resolving disagreements between the verifier and the task model\. No single configuration improves every metric monotonically, and the fullProcAgentachieves the strongest combined performance because each component is responsible for a different failure mode\.

![Refer to caption](https://arxiv.org/html/2607.24770v1/x6.png)Figure 6\.Hardware Performance and Thermal Analysis\. Comparative benchmarking of ablation configurations \(A–F\) and the integrated system \(ProcAgent\) on the NVIDIA Jetson AGX Orin platform\.\(A\)Power\-Efficiency Trade\-offs: Scatter plot showing the relationship between GPU load and power consumption\. The dashed red boundaries define a high thermal stress region \(Power ¿ 40W, Load ¿ 80%\), where configurations C and E reside\. The ”Full Sys” configuration achieves a balanced operating point, optimizing throughput without triggering thermal safety limits\.\(B\)Relative Computational Time Load: Horizontal bar chart displaying the normalized duration of processing sessions relative to the maximum recorded baseline \(Configuration C\)\. Configurations G and Full Sys demonstrate consistent temporal footprints \( 74%\) required for robust multimodal reasoning during procedural assembly\.\(C\)Thermal Profile Distribution: Mean GPU temperatures recorded across all experimental trials\. High\-performance configurations \(C, E\) approach the 60°C threshold, highlighting the necessity of the efficient resource management strategies employed in the finalized system \(55\.5°C\) to ensure long\-term hardware reliability\.
### 5\.5\.Resource and Thermal Profile

We characterize the on\-device resource cost of each configuration to evaluate whether the architectural choices identified in §[5\.3](https://arxiv.org/html/2607.24770#S5.SS3)translate into measurable runtime savings on the Jetson AGX Orin\. Three measurements are reported in Figure[6](https://arxiv.org/html/2607.24770#S5.F6): average GPU load against power consumption \(Figure[6](https://arxiv.org/html/2607.24770#S5.F6)a\), session duration normalized to the heaviest configuration \(Figure[6](https://arxiv.org/html/2607.24770#S5.F6)b\), and mean GPU temperature \(Figure[6](https://arxiv.org/html/2607.24770#S5.F6)c\)\.

Power and GPU Load\.Figure[6](https://arxiv.org/html/2607.24770#S5.F6)a shows that Configurations B and D \(visual verifier without task model\) sit inside the high\-thermal\-stress region, defined as average GPU load above 80% combined with average power above 40W\. In these configurations, every candidate emitted by the proposer is forwarded to the visual verifier without filtering\. Configurations that include the FSM \(C, E, F, andProcAgent\) sit outside this region\. The task model reduces verifier calls because candidates that do not match a guard\-satisfiable event from the current task state are filtered before verification, removing the GPU load associated with those calls\. The fullProcAgentoperates at roughly 75% GPU load and 41W average power, well within the safe operating envelope of the Jetson platform\.

Session Duration\.Figure[6](https://arxiv.org/html/2607.24770#S5.F6)b reports session duration as a percentage of Configuration B’s duration, the heaviest baseline\. Configurations with the task model are substantially shorter: Configuration C runs at 43\.8% of the baseline, Configuration E at 39\.1%, and the fullProcAgentat 74\.3%\. TheProcAgentfootprint is higher than the task model\-only configurations because user confirmation adds a confirmation\-dialogue turn to a subset of sessions, but it remains far below the task model\-excluded baselines\. Configurations without the task model \(B and D\) take roughly twice as long as their task model\-included counterparts, because each candidate requires a verifier call rather than being filtered upstream\.

Thermal Profile\.Figure[6](https://arxiv.org/html/2607.24770#S5.F6)c reports mean GPU temperature across all experimental trials\. Configurations B and D approach 59∘C, the highest temperatures in the study and within a few degrees of the thermal threshold beyond which the Jetson begins dynamic frequency scaling\. Configurations with the task model operate between 51∘C and 55∘C, and the fullProcAgentruns at 55\.5∘C\. Sustained operation in the high\-thermal regime would, over long sessions, trigger throttling and reduce inference throughput; the task model\-included configurations avoid this regime by construction\.

The resource profile supports the architectural argument made in §5\.4: the task model is not only a procedural\-control component but also a runtime efficiency mechanism\. Filtering candidates against guard conditions reduces verifier invocations, which in turn reduces GPU load, power draw, session duration, and thermal stress\. The fullProcAgentachieves the system’s procedural and quality wins while remaining within the Jetson’s safe operating envelope\.

## 6\.User Studies

To complement the quantitative ablation study, we conducted a user study to evaluate ProcAgent from the perspective of the people it is designed to help\. While the ablation study measures system\-level performance: state accuracy, sequence adherence, hallucination rate, the user study asks a different and equally important question: does the system actually help real users assemble furniture more successfully, and do they find it usable and trustworthy? This section describes the study design, tasks, participants, metrics, procedure, and results\.

### 6\.1\.Design

The study followed a within\-subjects design in which each participant completed an Lack Coffee Table assembly task withProcAgent\. Each session was conducted in a controlled indoor environment using the same hardware setup described in Section[5\.1\.1](https://arxiv.org/html/2607.24770#S5.SS1.SSS1): a Jetson AGX Orin running the full system pipeline, a Logitech C920s Pro camera, an EMEET Conference Speaker, and a directional microphone\. Participants were not shown any technical details of the system and were simply told they would be assembling furniture with the help of an AI assistant\.

### 6\.2\.Task

Participants were asked to assemble the Lack Coffee table IKEA furniture from scratch using only the physical components provided; no instruction manual was given\. The assembly task was selected from the same furniture categories used in the ablation study, covering a representative range of sub\-assembly complexity and step count\. The same task was used across all participants to ensure comparability of results\.

### 6\.3\.Participant

We recruited 10 participants \(2 female, 8 male\) through convenience sampling within our institution\. No participant had prior experience with AI\-assisted assembly tools\. All participants provided informed consent before the study began and were free to withdraw at any time\.

### 6\.4\.Metric

Post\-Session User Experience Survey\.After completing the session, each participant filled out a custom 8\-item post\-session survey evaluating their subjective experience withProcAgent\. Each item was rated on a 5\-point Likert scale ranging from Strongly Disagree to Strongly Agree, covering eight dimensions of the interaction experience:

- •\(Q1\) Comprehensibility: Whether the agent’s feedback was clear and easy to understand\.
- •\(Q2\) Actionability: Whether the instructions provided concrete, immediately actionable next steps\.
- •\(Q3\) Terminology: Whether the agent’s wording aligned naturally with furniture assembly language\.
- •\(Q4\) Confidence: Whether the agent’s feedback increased the participant’s confidence during assembly\.
- •\(Q5\) Mental Demand: Whether processing the agent’s feedback felt mentally demanding\.
- •\(Q6\) Responsiveness: Whether feedback was timely and responsive\.
- •\(Q7\) Privacy: Whether knowing that all processing was done locally on\-device made participants feel comfortable\.
- •\(Q8\) Trust: Whether participants trusted the agent’s proactive ”Hold on” interventions\.

### 6\.5\.Procedure

Each study session followed a structured protocol and lasted approximately 15 minutes in total\. Upon arrival, participants were given a brief overview of the study and a two\-minutes familiarization period during which the system’s interaction model was briefly explained, specifically, that they could ask the agent questions at any time and that the agent might occasionally speak to them unprompted if it detected something worth flagging\. No further instructions were given\. They were then introduced to the physical assembly components but given no instructions on how to proceed\.

After completing each session, participants filled out the user experience evaluation form\. Sessions were video recorded for post\-hoc analysis, and all recordings were stored locally on the study device in accordance with our institution’s data handling guidelines\. The study was approved by the authors’ institutional IRB \(Approval number: IRB\-25\-09024\-XP\), and all participants provided informed consent\.

### 6\.6\.Results

Post\-Session User Experience Survey\. The survey results, summarized in Figure[7](https://arxiv.org/html/2607.24770#S6.F7), reveal a consistently positive reception across most dimensions\. Comprehensibility \(Q1\) and Actionability \(Q2\) received the strongest endorsement, with all participants rating the agent’s feedback as clear and actionable, reflecting the effectiveness of the agent’s context\-aware, FSM\-grounded response generation in producing guidance that is specific and immediately usable rather than generic\.

Terminology alignment \(Q3\) and Privacy comfort \(Q7\) also received strong positive ratings, with the large majority of participants agreeing or strongly agreeing\. The strong Privacy result is particularly noteworthy: participants responded positively to knowing that all inference occurred locally on the device, suggesting that on\-device deployment is not merely a technical design choice but a meaningful contributor to user trust and comfort\.

![Refer to caption](https://arxiv.org/html/2607.24770v1/x7.png)Figure 7\.Likert\-scale user study results for ProcAgent \(n = 10\)\. After using the system, participants completed an 8\-item, 5\-point questionnaire\. The diverging stacked bar chart shows generally positive ratings across most usability and perception dimensions, with especially strong responses for comprehensibility, actionability, confidence, and privacy\.Confidence \(Q4\) and Trust in interventions \(Q8\) showed broadly positive results, though with slightly more neutral responses than Q1 and Q2\. This pattern is consistent with what one might expect from the first\-time users of an AI assembly assistant\. Participants were generally willing to follow the agent’s guidance but retained some residual uncertainty, particularly around the proactive interventions, which represent a novel and unfamiliar interaction pattern for most users\.

Mental Demand \(Q5\) was phrased as a negative item, meaning that disagreement reflects a positive outcome\. The strong skew toward Strongly Disagree indicates that the majority of participants did not find the system mentally taxing during assembly\. This validates two key design decisions: the agent’s deliberate restraint in speaking only when genuinely needed, and the Progressive Disclosure strategy of delivering a lightweight acknowledgment, accompanied by a brief waiting sound, before the full response\. Together, these kept cognitive load comfortably low even during the most demanding moments of the assembly task\.

Responsiveness and Feedback Time \(Q6\) received a broadly positive response, with most participants finding the system’s response timing acceptable\. The small number of neutral responses is understandable given that visually grounded responses can take 8\-12 seconds to generate\. The combination of an immediate verbal acknowledgment and a waiting sound appears to have effectively bridged this gap for most users, making the delay feel like a natural deliberative pause rather than a system lag\. That said, the neutral responses suggest that response latency remains perceptible to some users and is a worthwhile target for future optimization\.

## 7\.Discussion and Limitations

Our findings suggest thatProcAgentis best understood as a tightly coupled interactive system rather than a set of independent modules\. In the ablation, adding individual components sometimes reduced performance before the full pipeline was in place, indicating that incremental ablations can understate the value of interdependent mechanisms such as FSM tracking, visual verification, and human feedback\.

More broadly, the results show that human feedback serves as a primary source of state disambiguation rather than a fallback when perception fails\. User confirmations anchor the tracker to reliable ground truth and support more confident state transitions\. The gains in sequence adherence therefore reflect not only improved perception, but also the benefit of explicitly integrating user input into procedural tracking\.

The system also shows how edge deployment constraints can productively shape design\. Running on a single Jetson AGX Orin motivated staged verification, FSM\-based gating, and Progressive Disclosure, which together limit unnecessary model invocations and help manage latency\. At the same time, our results underscore that perception accuracy and procedural correctness are distinct: a system may recognize the current state correctly while still failing to guide the user through the correct next action\. In such settings, explicit task structure is as important as visual recognition\. The user study further suggests that trust depends not only on correctness, but also on selective intervention behavior\.

These findings should be interpreted in light of several limitations\. We evaluatedProcAgenton 15 demonstration videos across three IKEA furniture categories; broader evaluation on assemblies with longer horizons, greater branching, or more visually similar parts would better establish generality\. Although we use three LLM judges of varying capability, all three – GPT\-4o, GPT\-4\.1\-Mini, and GPT\-4\.1 – are from the same provider and share underlying training data, alignment procedures, and potential biases\. Scores that appear robust across the panel may still reflect systematic OpenAI\-family tendencies rather than true inter\-model independence\. Future evaluations should incorporate judges from distinct model families, such as Claude or Gemini, to provide stronger evidence of score robustness across genuinely diverse evaluators\. In the ablation, human feedback was simulated with a logic\-based trigger to isolate its functional role, but larger studies with real users would better capture variation in actual confirmations and mistakes\. The current FSTM supports flexible ordering among interchangeable sub\-steps but assumes forward progress and does not support undo or backward revision of previously confirmed steps\. ExtendingProcAgentto a new category also requires an offline authoring pipeline for step extraction, validation, and hierarchical context generation, which is amortized across sessions but still affects scalability\. Finally, the current implementation relies on a single fixed camera and assumes one active sub\-step at a time, leaving occlusion and concurrent sub\-step execution for future work\.

## 8\.Conclusion

We presentedProcAgent, a fully on\-device agentic framework for real\-time procedural assembly guidance, combining a conversational agent, a two\-stage perception stack, an FSM\-grounded task model, a RAG\-based knowledge base, and a Human\-in\-the\-Loop confirmation mechanism – all running locally on a single NVIDIA Jetson AGX Orin\. The central finding is that reliable procedural guidance requires more than accurate perception\. It requires knowing what is legally possible at each step, when visual verification is genuinely needed, and when to defer to the human\. No partial configuration achieves all three simultaneously – only the full system does, and the 33\-point adherence gain over the baseline reflects that\. Each component covers a distinct lackings of others: the Visual Verifier catches what the Low\-Latency Proposer misses, the FSM enforces what the Verifier cannot, RAG grounds what the FSM does not know, the query grounded visual context confirms what text alone cannot deliver, and Human Feedback resolves what no model can determine alone\.

User study participants responded positively across nearly all dimensions\. Feedback was rated as clear and immediately actionable by all participants, and the system’s use of assembly\-specific terminology felt natural\. Privacy comfort was notably strong – participants responded positively to knowing all inference ran locally, suggesting on\-device deployment contributes to trust beyond its technical merits\. Confidence and trust in the system’s proactive interventions were broadly positive, with some residual uncertainty expected from first\-time users encountering a novel interaction pattern\. Critically, participants did not find the system mentally taxing, a direct validation of the agent’s deliberate restraint and the Progressive Disclosure strategy, which together kept cognitive load low throughout the task\. Response latency was acceptable to most users, with the immediate verbal acknowledgment and waiting sound effectively masking the 8–12 second VLM verification delay for the majority of participants, though it remains a target for future optimization\.

Future work should focus on broader task domains, multi\-camera setups to address occlusion, backward revision of completed steps, and reduced VLM latency\.ProcAgentultimately shows that useful AI assistance in the physical world does not require the cloud – it requires the right design\.

## References

- M\. Ahn, A\. Brohan, N\. Brown, Y\. Chebotar, O\. Cortes, B\. David, C\. Finn, C\. Fu, K\. Gopalakrishnan, K\. Hausman, A\. Herzog, D\. Ho, J\. Hsu, J\. Ibarz, B\. Ichter, A\. Irpan, E\. Jang, R\. J\. Ruano, K\. Jeffrey, S\. Jesmonth, N\. J\. Joshi, R\. Julian, D\. Kalashnikov, Y\. Kuang, K\. Lee, S\. Levine, Y\. Lu, L\. Luu, C\. Parada, P\. Pastor, J\. Quiambao, K\. Rao, J\. Rettinghouse, D\. Reyes, P\. Sermanet, N\. Sievers, C\. Tan, A\. Toshev, V\. Vanhoucke, F\. Xia, T\. Xiao, P\. Xu, S\. Xu, M\. Yan, and A\. Zeng \(2022\)Do as i can, not as i say: grounding language in robotic affordances\.External Links:2204\.01691,[Link](https://arxiv.org/abs/2204.01691)Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p1.1)\.
- M\. Ahn, D\. Dwibedi, C\. Finn, M\. G\. Arenas, K\. Gopalakrishnan, K\. Hausman, B\. Ichter, A\. Irpan, N\. Joshi, R\. Julian, S\. Kirmani, I\. Leal, E\. Lee, S\. Levine, Y\. Lu, I\. Leal, S\. Maddineni, K\. Rao, D\. Sadigh, P\. Sanketi, P\. Sermanet, Q\. Vuong, S\. Welker, F\. Xia, T\. Xiao, P\. Xu, S\. Xu, and Z\. Xu \(2024\)AutoRT: embodied foundation models for large scale orchestration of robotic agents\.External Links:2401\.12963,[Link](https://arxiv.org/abs/2401.12963)Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p3.1)\.
- R\. Arakawa, J\. F\. Lehman, and M\. Goel \(2024\)PrISM\-q&a: step\-aware voice assistant on a smartwatch enabled by multimodal procedure tracking and large language models\.Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\.8\(4\)\.External Links:[Link](https://doi.org/10.1145/3699759),[Document](https://dx.doi.org/10.1145/3699759)Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p2.1)\.
- Y\. Ben\-Shabat, X\. Yu, F\. Saleh, D\. Campbell, C\. Rodriguez\-Opazo, H\. Li, and S\. Gould \(2021\)The ikea asm dataset: understanding people assembling furniture through actions, objects and pose\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 847–859\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p4.1),[§5\.1\.2](https://arxiv.org/html/2607.24770#S5.SS1.SSS2.p1.1),[§5\.1](https://arxiv.org/html/2607.24770#S5.SS1.SSSx1.p2.1)\.
- M\. Bock, H\. Kuehne, K\. Van Laerhoven, and M\. Moeller \(2024\)Wear: an outdoor sports dataset for wearable and egocentric activity recognition\.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies8\(4\),pp\. 1–21\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p2.1)\.
- W\. Dai, J\. Li, D\. Li, A\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. N\. Fung, and S\. Hoi \(2023\)Instructblip: towards general\-purpose vision\-language models with instruction tuning\.Advances in neural information processing systems36,pp\. 49250–49267\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p4.1)\.
- D\. Damen, H\. Doughty, G\. M\. Farinella, S\. Fidler, A\. Furnari, E\. Kazakos, D\. Moltisanti, J\. Munro, T\. Perrett, W\. Price,et al\.\(2018\)Scaling egocentric vision: the epic\-kitchens dataset\.InProceedings of the European conference on computer vision \(ECCV\),pp\. 720–736\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p2.1)\.
- P\. Dourish \(2004\)What we talk about when we talk about context\.Personal and ubiquitous computing8\(1\),pp\. 19–30\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1)\.
- A\. K\. Eesee, V\. Varga, G\. Eigner, and T\. Ruppert \(2025\)Impact of work instruction difficulty on cognitive load and operational efficiency\.Scientific Reports15\(1\),pp\. 11028\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p1.1)\.
- EMEET \(2020\)EMEET officecore luna conference speakerphone\.Note:Accessed 2026\-05\-02External Links:[Link](https://emeet.com/products/speakerphone-luna)Cited by:[§5\.1\.1](https://arxiv.org/html/2607.24770#S5.SS1.SSS1.p1.1)\.
- X\. Gao, Y\. Jin, Y\. Long, Q\. Dou, and P\. Heng \(2021\)Trans\-svnet: accurate phase recognition from surgical videos via hybrid embedding aggregation transformer\.InInternational conference on medical image computing and computer\-assisted intervention,pp\. 593–603\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p2.1)\.
- H\. A\. Hartanto, I\. K\. Makrie, V\. Yesmaya, and P\. G\. Halfian \(2019\)The development of furniture assembly instruction based on augmented\-reality\.ComTech: Computer, Mathematics and Engineering Applications10\(2\),pp\. 75–81\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p1.1)\.
- K\. Hasegawa, W\. Imrattanatrai, M\. Asada, S\. Holm, Y\. Wang, V\. Zhou, K\. Fukuda, and T\. Mitamura \(2025\)ProMQA\-assembly: multimodal procedural qa dataset on assembly\.arXiv preprint arXiv:2509\.02949\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1)\.
- P\. Hegemann, T\. Zechmeister, M\. Grotz, K\. Hitzler, and T\. Asfour \(2022\)Learning symbolic failure detection for grasping and mobile manipulation tasks\.In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 4302–4309\.Cited by:[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p1.1)\.
- Y\. Hu, F\. Lin, T\. Zhang, L\. Yi, and Y\. Gao \(2023\)Look before you leap: unveiling the power of gpt\-4v in robotic vision\-language planning\.arXiv preprint arXiv:2311\.17842\.Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p1.1)\.
- J\. Huang, Y\. Jin, L\. An, and J\. Park \(2025\)LiteVLM: a low\-latency vision\-language model inference pipeline for resource\-constrained environments\.arXiv preprint arXiv:2506\.07416\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p3.1)\.
- S\. Kletz, K\. Schoeffmann, J\. Benois\-Pineau, and H\. Husslein \(2019\)Identifying surgical instruments in laparoscopy using deep learning instance segmentation\.In2019 International Conference on Content\-Based Multimedia Indexing \(CBMI\),pp\. 1–6\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p3.1)\.
- \[18\]W\. Kryściński, B\. McCann, C\. Xiong, and R\. SocherEvaluating the factual consistency of abstractive text summarization\. arxiv 2019\.arXiv preprint arXiv:1910\.12840\.Cited by:[§5\.1](https://arxiv.org/html/2607.24770#S5.SS1.SSSx3.p2.1)\.
- C\. Lea, M\. D\. Flynn, R\. Vidal, A\. Reiter, and G\. D\. Hager \(2017\)Temporal convolutional networks for action segmentation and detection\.Inproceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 156–165\.Cited by:[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p2.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p3.1)\.
- Logitech \(2026\)Logitech c920s pro full hd webcam with privacy shutter\.Note:Accessed 2026\-05\-02External Links:[Link](https://www.logitech.com/en-us/shop/p/c920s-pro-hd-webcam)Cited by:[§5\.1\.1](https://arxiv.org/html/2607.24770#S5.SS1.SSS1.p1.1)\.
- W\. Luo, D\. Zhang, Y\. Tang, F\. Wu, and Y\. Zhang \(2025\)EdgeOAR: real\-time online action recognition on edge devices\.IEEE Transactions on Mobile Computing\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1)\.
- H\. Meng, M\. Freeman, N\. Pears, and C\. Bailey \(2008\)Real\-time human action recognition on an embedded, reconfigurable video processing architecture\.Journal of Real\-Time Image Processing3\(3\),pp\. 163–176\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1)\.
- NVIDIA \(2021\)NVIDIA jetson agx orin series\.Technical reportNVIDIA\.External Links:[Link](https://www.nvidia.com/content/dam/en-zz/Solutions/gtcf21/jetson-orin/nvidia-jetson-agx-orin-technical-brief.pdf)Cited by:[§5\.1\.1](https://arxiv.org/html/2607.24770#S5.SS1.SSS1.p1.1)\.
- R\. Peddi, S\. Arya, B\. Challa, L\. Pallapothula, A\. Vyas, B\. Gouripeddi, Q\. Zhang, J\. Wang, V\. Komaragiri, E\. Ragan,et al\.\(2024\)Captaincook4d: a dataset for understanding errors in procedural activities\.Advances in Neural Information Processing Systems37,pp\. 135626–135679\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1)\.
- X\. Qin, S\. Luan, J\. See, C\. Yang, and Z\. Li \(2026\)Harnessing embodied agents: runtime governance for policy\-constrained execution\.arXiv preprint arXiv:2604\.07833\.Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p3.1)\.
- R\. Radkowski, J\. Herrema, and J\. Oliver \(2015\)Augmented reality\-based manual assembly support with visual features for different degrees of difficulty\.International Journal of Human\-Computer Interaction31\(5\),pp\. 337–349\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p1.1)\.
- P\. Ramjee, B\. Sachdeva, S\. Golechha, S\. Kulkarni, G\. Fulari, K\. Murali, and M\. Jain \(2025\)CataractBot: an llm\-powered expert\-in\-the\-loop chatbot for cataract patients\.Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\.9\(2\)\.External Links:[Link](https://doi.org/10.1145/3729479),[Document](https://dx.doi.org/10.1145/3729479)Cited by:[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p3.1)\.
- G\. M\. Re, J\. Oliver, and M\. Bordegoni \(2016\)Impact of monitor\-based augmented reality for on\-site industrial manual operations\.Cognition, Technology & Work18\(2\),pp\. 379–392\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p1.1)\.
- G\. Sarch, Y\. Wu, M\. Tarr, and K\. Fragkiadaki \(2023\)Open\-ended instructable embodied agents with memory\-augmented large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 3468–3500\.Cited by:[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p4.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.External Links:2302\.04761,[Link](https://arxiv.org/abs/2302.04761)Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p1.1)\.
- F\. Sener, D\. Chatterjee, D\. Shelepov, K\. He, D\. Singhania, R\. Wang, and A\. Yao \(2022\)Assembly101: a large\-scale multi\-view video dataset for understanding procedural activities\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 21096–21106\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p2.1)\.
- J\. L\. Sewell, C\. K\. Boscardin, J\. Q\. Young, O\. Ten Cate, and P\. S\. O’Sullivan \(2016\)Measuring cognitive load during procedural skills training with colonoscopy as an exemplar\.Medical education50\(6\),pp\. 682–692\.Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p1.1)\.
- D\. Shao, Y\. Zhao, B\. Dai, and D\. Lin \(2020\)Finegym: a hierarchical video dataset for fine\-grained action understanding\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 2616–2625\.Cited by:[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p2.1)\.
- Z\. Shen, Y\. He, Z\. Wang, Y\. Zhang, G\. Sun, W\. Ye, and A\. Li \(2025\)EdgeLoRA: an efficient multi\-tenant llm serving system on edge devices\.InProceedings of the 23rd Annual International Conference on Mobile Systems, Applications and Services,MobiSys ’25,New York, NY, USA,pp\. 138–153\.External Links:ISBN 9798400714535,[Link](https://doi.org/10.1145/3711875.3729141),[Document](https://dx.doi.org/10.1145/3711875.3729141)Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p4.1)\.
- C\. Sun, J\. Han, W\. Deng, X\. Wang, Z\. Qin, and S\. Gould \(2025\)3d\-gpt: procedural 3d modeling with large language models\.In2025 International Conference on 3D Vision \(3DV\),pp\. 1253–1263\.Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p2.1)\.
- B\. J\. Tang, K\. Sun, N\. T\. Curran, F\. Schaub, and K\. G\. Shin \(2025\)Ads that talk back: implications and perceptions of injecting personalized advertising into llm chatbots\.Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\.9\(4\)\.External Links:[Link](https://doi.org/10.1145/3770640),[Document](https://dx.doi.org/10.1145/3770640)Cited by:[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p3.1)\.
- S\. Tuli, G\. Casale, and M\. Roveri \(2025\)SELA: smart edge llm agent to optimize response trade\-offs of ai assistants\.Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\.9\(3\)\.External Links:[Link](https://doi.org/10.1145/3749483),[Document](https://dx.doi.org/10.1145/3749483)Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p4.1)\.
- R\. Wang, P\. Gao, P\. Lynch, T\. Liu, Y\. Lee, C\. Baum, L\. T\. Connor, and C\. Lu \(2025\)CHEF\-vl: detecting cognitive sequencing errors in cooking with vision\-language models\.Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\.9\(4\)\.External Links:[Link](https://doi.org/10.1145/3770714),[Document](https://dx.doi.org/10.1145/3770714)Cited by:[§1](https://arxiv.org/html/2607.24770#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.24770#S2.SS2.p2.1)\.
- X\. Wang, T\. Kwon, M\. Rad, B\. Pan, I\. Chakraborty, S\. Andrist, D\. Bohus, A\. Feniello, B\. Tekin, F\. V\. Frujeri,et al\.\(2023\)Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20270–20281\.Cited by:[§2\.1](https://arxiv.org/html/2607.24770#S2.SS1.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p1.1)\.
- Y\. Zhang, X\. Liu, and C\. Xiao \(2025\)Metaagent: automatically constructing multi\-agent systems based on finite state machines\.arXiv preprint arXiv:2507\.22606\.Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p3.1)\.
- A\. Y\. Zhao, A\. Gunturu, E\. Y\. Do, and R\. Suzuki \(2025\)Guided reality: generating visually\-enriched ar task guidance with llms and vision models\.InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology,pp\. 1–15\.Cited by:[§2\.3](https://arxiv.org/html/2607.24770#S2.SS3.p2.1)\.

## Appendix AMulti\-Judge LLM Evaluation Prompt

Table 7\.You are an expert evaluator for an AI\-assisted furniture assembly system\. Your task is to evaluate the quality of an AI assistant’s response given:1\. A knowledge base \(KB\) describing the assembly process2\. The current assembly state3\. A user question4\. The assistant‘s answerINPUTSKnowledge Base \(KB\): \{knowledge\_base\}Ground Truth Sequence: \{ground\_truth\_sequence\}Task Sequence \(may be incomplete or user\-facing version\): \{task\_sequence\}Rules for Completion: \{rules\_for\_completion\}Current Assembly State: \{current\_state\}User Question: \{question\}Assistant Answer: \{answer\}EVALUATION CRITERIAEvaluate the assistant‘s answer along the following three axes:1\. Hallucination \(0–5\)Does the answer introduce information that is NOT supported by the KB or current state?5 → No hallucination \(fully grounded in KB/state\)4 → Minor unsupported assumptions, but mostly grounded3 → Some unsupported or speculative content2 → Significant hallucination affecting reliability1 → Mostly hallucinated0 → Completely fabricated / unrelated2\. Correctness \(0–5\)Is the answer factually correct and consistent with the KB and current state?5 → Fully correct and consistent4 → Minor inaccuracies but overall correct3 → Partially correct with notable mistakes2 → Mostly incorrect1 → Largely incorrect0 → Completely wrong3\. Helpfulness \(0–5\)Does the answer effectively help the user progress in assembly?5 → Clear, actionable, and directly helps progress4 → Helpful but slightly incomplete or unclear3 → Somewhat helpful but lacks clarity or steps2 → Minimally helpful1 → Confusing or not actionable0 → Not helpful at allIMPORTANT INSTRUCTIONSGround your evaluation strictly in the provided KB and current state\.Do NOT assume missing steps unless clearly implied\.Penalize hallucinations even if the answer sounds plausible\.Consider whether the answer is appropriate for the “current assembly stage”\.OUTPUT FORMAT \(STRICT JSON\)\{\{”hallucination\_score”: <0\-5\>,”correctness\_score”: <0\-5\>,”helpfulness\_score”: <0\-5\>\}\}

## Appendix BHyper\-parameter Search

Table 8\.
## Appendix CAtomic Steps Extraction Prompt

Table 9\.ROLEAct as an expert Furniture Assembly Action Classifier\.VIDEO SPECIFICATIONSDuration: 180 secondsFrame Rate: 30 FPSProcessing Logic:Every 1 second of video equals 30 frames\. Use this linear scale to ensure timestamps and frame indices do not drift over the 3\-minute duration\.TASKAnalyze the provided video and extract a sequence of atomic steps\.CONSTRAINTS1\. ONLY use the following standardized action names \(Verb\-Noun pairs\):\- Verbs: \[pick up, align, spin, attach, flip, insert, tighten, remove, align and spin\]\- Nouns: \[leg, shelf, table, screw, base, panel, drawer, frame\]2\. Cardinality Rule: Log EVERY instance of a repetitive action\. Do NOT summarize\.3\. If an action does not fit the list, use the label ”others”\.4\. Follow the Step Extraction Protocol: split complex movements into atomic steps\.OUTPUT FORMAT \(JSON\)Return ONLY the JSON\.\{”reference\_video\_id”: ”\{\{job\_id\}\}”,”furniture\_name”: ”…”,”metadata”: \{”fps”: 30,”video\_duration\_seconds”: 180\},”steps”: \[\{”name”: ”verb \+ noun”,”timestamp”: ”MM:SS”,”frame\_index”: 0000,”description”: ”Short visual anchor \(e\.g\., hand touches leg\)”\}\]\}

## Appendix DVideo Dataset for Ablation Studies

Table 10\.List of 15 Video Sequences used in Ablation Studies