Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints

# Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints
Source: [https://arxiv.org/html/2608.13684](https://arxiv.org/html/2608.13684)
Alex LascaridesEmail:[alex@inf\.ed\.ac\.uk](mailto:[email protected])Subramanian RamamoorthyEmail:[s\.ramamoorthy@ed\.ac\.uk](mailto:[email protected])Affiliation:School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK

###### Abstract

This paper describes a neurosymbolic architecture for learning to assemble novel structures using evidence from embodied conversations and task demonstrations\. We focus on scenarios where an agent encounters, after deployment,semantic constraintson structures—in other words, constraints as to which part types and features make valid structures—that were not available during training, and where it isinitially unawareof the relevant structure and component part concepts\. The agent must acquire and exploit such knowledge through user interactionswhileattempting assembly\. We study this setting in a simulated toy truck assembly domain, learning from symbolic evidence encoded in natural language and from dense visual observations\. Our experiments show that communicating semantic constraints through natural language \(e\.g\., “dump trucks have a dumper”\) yields more data\-efficient online adaptation than relying only on task demonstrations and/or only naming the parts through natural language\.

## 1Introduction

Robotic assembly in open\-ended environments requires agents to cope with task knowledge that may not be available before deployment\([5](https://arxiv.org/html/2608.13684#bib.bib5)\)\. Prior work commonly distinguishes assembly knowledge into i\) geometry of parts and joins, ii\) assembly orders, and iii\) low\-level control skills\([9](https://arxiv.org/html/2608.13684#bib.bib9)\)\. These aspects comprise largely physical constraints and are well suited to learning from task demonstrations\([22](https://arxiv.org/html/2608.13684#bib.bib22)\)\.

However, successful assembly may also depend onsemantic constraints: arbitrary domain conventions under which it is physically feasible to join two parts, but doing so creates an invalid structure\. In the example shown in Fig\.[1](https://arxiv.org/html/2608.13684#S1), the join attempt is corrected despite geometric compatibility because it violates a semantic rule: namely that “Coloured parts of a dump truck must not be yellow”\. Demonstrations alone are insufficient as a corrective signal here, since multiple rules may be consistent with them: the constraint in Fig\.[1](https://arxiv.org/html/2608.13684#S1)might have been, for example, “Dump trucks must have a red chassis center”\. The scope of the rule is also latent, as it may apply to a specific subtype of truck or to all trucks\. The challenge becomes even greater when the agent is initially unaware of the concepts in which the constraints are expressed; e\.g\., if it never encountered a flat chassis center nor its label during training\. The agent cannot logically infer a rule referring to a concept that it doesn’t know exists\.

\\subfigure\[Truck types defined by parts\.\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/truck_domain.png)\\subfigure\[User\-agent dialogue\.\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/opening_example.png)

We address this setting with a neurosymbolic agent architecture for online adaptation from embodied natural language \(NL\) interaction and task demonstrations\. The central challenge is that domain knowledge may be introduced piecemeal during deployment, and the agent must change its behaviour on the timescale of the current interaction\. This motivates separating dense perceptual grounding from explicit symbolic memory and planning: visual classifiers can be updated via new exemplars; the lexicon and knowledge base can be extended with new concepts and constraints; and the planner can immediately use the updated knowledge to revise object selection and action sequences\. The neurosymbolic coupling is therefore primarily perception\-to\-reasoning: neural perception supplies graded candidate groundings, and newly acquired symbolic knowledge selects among these groundings and revises action plans, without directly revising the visual feature representation\. This design targets the online adaptation problem studied here, while leaving tighter symbolic\-to\-neural feedback as a compatible extension for settings that require representation\-level revision\.

We evaluate the proposed framework in a simulated toy truck assembly domain\. Truck types are defined by constraints over part subtypes and attributes \(see Fig\.[1](https://arxiv.org/html/2608.13684#S1)\)\. We compare an agent that can interpret generic NL statements expressing semantic constraints—such asdump trucks have a dumper—against ablative baselines that receive demonstrations, part labels, and/or non\-generic correction signals\. The results show that communicating semantic constraints through NL generics improves data efficiency for online learning in this assembly domain, beyond the benefits of naming novel part concepts alone\.

## 2Semantic Assembly: Task Setting and Domain

### 2\.1Task Domain, Goal and Input Formulation

In our formulation, a semantic assembly domain defines two sets of concepts,Π\\PiandΣ\\Sigma:

- •Π\\Piis a set ofatomic part concepts, represented as unary predicates over primitive objects; their instances are minimal building blocks of assembly structures: e\.g\.,dumper,quad\_cabin,cabin\.
- •Σ\\Sigmais a set ofsubassembly concepts, represented as unary predicates over composite structures; their instances consist of more than one atomic parts assembled together: e\.g\.,truck,fire\_truck,truck\_front\.

Hyper/hyponymy, oris\-a, relations are represented as subclass implications between unary predicates inΠ\\Piand inΣ\\Sigma: for instance,quad\_cabinis a subtype ofcabin; andfire\_truckis a subtype oftruck\. Holo/meronymy, orhas\-a, relations are represented using the binary predicateh​a​v​e​\(x,y\)have\(x,y\)betweenΠ\\PiandΣ\\Sigma\(e\.g\., Atruckhas acabin\) or withinΣ\\Sigma\(e\.g\., Atruckhas atruck\_front\)\. Together, these predicates and implication/relation rules constitute the domain theoryΩ\\Omega\.

Each instance of a semantic assembly task is characterised by: \(a\) two sets of objectsCCandDDlaid out on the tabletop; and \(b\) an assembly goalγ∈Σ\\gamma\\in\\Sigma\.CCcontains exactly the ‘ground\-truth’ atomic parts required to build an instance ofγ\\gamma, whileDDis a \(possibly empty\) set of ‘distractor’ parts, which are geometrically feasible for assembly but will violate a semantic constraint if used\. Given the subtype rules withinΠ\\Pi, an object inCCorDDmay satisfy multiple part predicates\. Parts also have colour attributes, and so may be referred to as ‘a red quad cabin’ in NL, for example\.

Each goal conceptγ∈Σ\\gamma\\in\\Sigmais associated with two formal specificationsGγG\_\{\\gamma\}andSγS\_\{\\gamma\}, which together define the range of valid assembly structures admissible as an instance ofγ\\gamma:

- •Gγ=\(Vγ,Eγ\)G\_\{\\gamma\}=\(V\_\{\\gamma\},E\_\{\\gamma\}\)is an intensional structural specification \(‘assembly topology graph’\) ofγ\\gammathat specifies the necessary and sufficient part joins in a valid instance ofγ\\gamma\. Each node inVγV\_\{\\gamma\}specifies a type requirement fromΠ\\PiorΣ\\Sigma\. The edgesEγE\_\{\\gamma\}specify required joins and their relative poses\.
- •SγS\_\{\\gamma\}is a set of quantified first\-order logic \(FOL\) rules or integrity constraints relevant toγ\\gamma\(see Fig\.[1](https://arxiv.org/html/2608.13684#S1)\), which must hold in all instances ofγ\\gamma\. For instance, “Dump trucks have a dumper” is encoded as∀x​∃y\.d​u​m​p​T​r​u​c​k​\(x\)→d​u​m​p​e​r​\(y\)∧h​a​v​e​\(x,y\)\\forall x\\exists y\.dumpTruck\(x\)\\rightarrow dumper\(y\)\\land have\(x,y\)\.

The task objective is to plan and execute a sequence of actions that constructs a structure satisfyingGγG\_\{\\gamma\}while observing all constraints inSγS\_\{\\gamma\}, using only parts fromCCand notDD\. The agent has a single\-view RGB imageℐ∈\[0,1\]3×H×W\\mathcal\{I\}\\in\[0,1\]^\{3\\times H\\times W\}of the scene, whereHHandWWare the sensor height and width respectively\. During assembly, the agent engages in an NL dialogue with a teacher who supervises the task execution; see §[2\.4](https://arxiv.org/html/2608.13684#S2.SS4)for details\.

### 2\.2Scope and Assumptions

In this study, we make several simplifying assumptions to isolate deployment\-time acquisition of semantic assembly knowledge from low\-level robotics and open\-domain language\-understanding challenges\. Experiments are conducted in a simulated toy\-truck domain and we evaluate online adaptation within this domain, rather than cross\-domain transfer or real\-world robustness\. The agent receives oracle object masks and 6D poses from the simulator\. Demonstrations are assumed to be segmented into parametrised primitive actions which are executed by an oracle controller\. Teacher interactions follow a controlled dialogue protocol, so the experiments test the effect of receiving generic semantic content rather than open\-domain NL understanding\. The visual feature backbone is fixed: online perceptual adaptation updates exemplar memories and concept classifiers, not visual features\. Colour predicates are grounded by the same exemplar\-based procedure as part concepts, but their classifiers are initialised before the main assembly episodes from a small set of positive and negative exemplars\. Finally, instances of the same atomic part concept are assumed to share a 3D geometry, enabling type\-level point\-cloud and join\-pose representations\.

### 2\.3Initial Agent Knowledge and Task Abstraction

We consider a setting in which the agent possesses domain\-neutral skills but lacks domain\-specific assembly knowledge\. The domain\-neutral skills include primitive action interfaces and a high\-level symbolic planning faculty that deploys them\. We adoptAnswer Set Programming\(ASP;[10](https://arxiv.org/html/2608.13684#bib.bib10)\), a declarative programming approach based on normal logic programs\. We use ASP because it allows seamless integration of learned semantic constraints in planning, also encoded as ASP program fragments, thanks to its ability to model indirect action effects\([20](https://arxiv.org/html/2608.13684#bib.bib20)\)\. App\.[A](https://arxiv.org/html/2608.13684#A1)describes how the assembly planning problem is implemented as ASP programs\.

At the start, the agent is ignorant of domain\-specific knowledge: the concept setsΠ\\PiandΣ\\Sigma\(i\.e\., the hypothesis space of possible parts and structures\), the domain theoryΩ\\Omega, the 2D\-visual and 3D\-geometric features of the atomic parts inΠ\\Pi, the assembly topology graphsGγG\_\{\\gamma\}and the semantic constraintsSγS\_\{\\gamma\}for eachγ∈Σ\\gamma\\in\\Sigmaare empty\. The agent thus lacks the hypothesis space of possibilities and must acquire it after deployment\. Our main hypothesis is that generic NL statements improve online learning by communicating reusable semantic constraints, beyond what can be obtained from demonstrations and part labels alone\.

### 2\.4Agent\-Teacher Interactions

Task demonstrations and NL dialogue provide complementary evidence\. Demonstrations convey continuous information, in particular the relative poses required to join parts\. Within our controlled dialogue protocol, each episode begins with a teacher\-specified goal: “Build aγ\\gamma”\. If the agent lacks the target concept or required part concepts, the teacher provides either a demonstration, a verbal definition, or labelled exemplars\. During planning and execution, agent failures expose what kind of knowledge is missing: an “Is there a dumper?” query calls for a labelled exemplar; a mistaken “I meant to join this quad cabin” response lets the teacher correct the part type; and a geometrically feasible but invalid join calls for a semantic constraint\. Teacher feedback targets the exposed gap, updating visual exemplars, the lexicon, or symbolic memory before replanning\. A fuller account of the possible interaction flows, along with a visual flowchart depiction, is provided in App\.[B](https://arxiv.org/html/2608.13684#A2)\.

## 3The Neurosymbolic Agent Architecture

This section describes the neurosymbolic architecture used to implement the interaction protocol above\. Before describing individual modules, Alg\.[1](https://arxiv.org/html/2608.13684#alg1)summarises the episode\-level control loop: perception produces candidate symbolic groundings, dialogue updates memory when knowledge gaps are exposed, symbolic reasoning selects goals and plans joins, and actuation executes the selected actions\. Fig\.illustrates the overall architecture\.

Algorithm 1Interactive semantic assembly episode1:goal utterance

uu, scene image

ℐ\\mathcal\{I\}, visual exemplar base

X​BXB, symbolic knowledge base

K​B=\(Ω,\{Gσ,Sσ\}σ∈Σ\)KB=\(\\Omega,\\\{G\_\{\\sigma\},S\_\{\\sigma\}\\\}\_\{\\sigma\\in\\Sigma\}\), lexicon

LL
2:Map

uuto target concept

γ\\gammausing lexicon

LL\.

3:if

γ\\gammaor required concepts are unknownthen

4:Obtain teacher definition, labelled exemplars, or demonstration\.

5:Update

X​BXB,

K​BKB,

LLas applicable\.

6:endif

7:whiletask not accepted by teacherdo

8:Perceive scene from

ℐ\\mathcal\{I\}and produce candidate part groundings with scores\.

9:Solve goal\-selection ASP to choose an object\-filled goal structure under current

GγG\_\{\\gamma\},

SγS\_\{\\gamma\}, and grounding\.

10:ifa required part cannot be groundedthen

11:Ask teacher for an instance and update

X​BXB\.

12:continue

13:endif

14:Solve join\-sequence ASP using the selected goal structure, then execute the resulting sequence\.

15:ifteacher interruptsthen

16:Explain intended action\.

17:Update

X​BXBand/or

K​BKBaccording to the diagnosed error and active strategy\.

18:endif

19:endwhile

### 3\.1Vision Processing Module

The vision processing module maps dense visual inputs to structured symbolic representations consumed by the planning and learning components\. Given a single\-view RGB image of the scene, the module outputs, for each detected object instance: 1\) a set of candidate atomic part concepts fromΠ\\Piwith associated confidence scores; and 2\) a 3D geometric representation suitable for specifying join relations\. Confidence scores produced by the vision module are propagated to the symbolic layer, where they influence planning decisions\. Implementation details of feature extraction, classification, geometry extraction, and pose handling are given in App\.[C](https://arxiv.org/html/2608.13684#A3)\.

![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/architecture.png)
### 3\.2Dialogue Management Module

The dialogue management module interprets teacher utterances and generates agent responses in the controlled interaction protocol of §[2\.4](https://arxiv.org/html/2608.13684#S2.SS4)\. It handles goal utterances, clarification requests, corrective feedback, and generic statements that express semantic constraints\. Our implementation deploys an off\-the\-shelf large\-coverage semantic parser\([2](https://arxiv.org/html/2608.13684#bib.bib2)\), appended with a heuristic postprocessing pipeline\. Here, generic statements that express semantic constraints take two forms:

- •Universal constraints dictate that a part type of a subassembly must all have certain qualities\. For example, “All fenders of fire trucks are red” is: ∀x​∀y\.f​i​r​e​T​r​u​c​k​\(x\)∧f​e​n​d​e​r​\(y\)∧h​a​v​e​\(x,y\)→r​e​d​\(y\)\\displaystyle\\forall x\\forall y\.fireTruck\(x\)\\land fender\(y\)\\land have\(x,y\)\\rightarrow red\(y\)
- •Existential constraints, that a certain subassembly must have at least one part with certain qualities\. For example, “Missile trucks have a green fender\.”: ∀x​∃y\.m​i​s​s​i​l​e​T​r​u​c​k​\(x\)→f​e​n​d​e​r​\(y\)∧g​r​e​e​n​\(y\)∧h​a​v​e​\(x,y\)\\displaystyle\\forall x\\exists y\.missileTruck\(x\)\\rightarrow fender\(y\)\\land green\(y\)\\land have\(x,y\)

These FOL statements undergo an additional automated translation step into ASP program clauses, to be integrated with ASP planning problems\. The above examples become the following, wherecic\_\{i\}is the index of the violated constraint:

v​i​o​l​a​t​e​d​\(ci\)←f​i​r​e​T​r​u​c​k​\(X\),f​e​n​d​e​r​\(Y\),h​a​v​e​\(X,Y\),not​r​e​d​\(Y\)\.violated\(c\_\{i\}\)\\leftarrow fireTruck\(X\),fender\(Y\),have\(X,Y\),\\text\{ not \}red\(Y\)\.

violated\(ci\)←missileTruck\(X\),\#count\{Y:fender\(Y\),green\(Y\),have\(X,Y\)\}=0violated\(c\_\{i\}\)\\leftarrow missileTruck\(X\),\\\#count\\\{Y:fender\(Y\),green\(Y\),have\(X,Y\)\\\}=0\.

### 3\.3Actuation Module

The actuation module executes the primitive actions selected by the symbolic planner using the oracle simulator controller described in §[2\.2](https://arxiv.org/html/2608.13684#S2.SS2)\.

### 3\.4Long\-term Memory Module

The long\-term memory module stores the knowledge updated through Alg\.[1](https://arxiv.org/html/2608.13684#alg1): the visual exemplar base \(XB\), the symbolic knowledge base \(KB\), and the lexicon\. The visual XB stores positive and negative visual exemplars for each atomic part conceptπ∈Π\\pi\\in\\Pi, respectivelyχπ\+\\chi^\{\+\}\_\{\\pi\}andχπ−\\chi^\{\-\}\_\{\\pi\}\. Each exemplar is represented as a feature vector generated by the vision processing module\. Whenever eitherχπ\+\\chi^\{\+\}\_\{\\pi\}orχπ−\\chi^\{\-\}\_\{\\pi\}is updated, a new binary classifier forπ\\piis induced\. New entries are added when the teacher says an object is \(not\) an example ofπ\\pi\.

The symbolic KB stores the domain theoryΩ\\Omega, the assembly topology graphsGγG\_\{\\gamma\}, and the semantic constraint setsSγS\_\{\\gamma\}for eachγ∈Σ\\gamma\\in\\Sigma\. As explained in §[2\.4](https://arxiv.org/html/2608.13684#S2.SS4)and §[3\.2](https://arxiv.org/html/2608.13684#S3.SS2), these structures are populated from teacher definitions, demonstrations, and generic utterances\.

The lexicon maps concepts inΠ\\Pi,Σ\\Sigmato their NL labels\. We do not directly use NL labels as the internal indicesπ∈Π\\pi\\in\\Pi; instead, the lexicon tracks associations between internal concept indices and teacher\-provided labels\. This separation allows us to compare strategies with and without shared part vocabularies\.

### 3\.5Symbolic Reasoning Module

The symbolic reasoning module implements the two ASP planning subproblems in Alg\.[1](https://arxiv.org/html/2608.13684#alg1): goal selection and join\-sequence planning\. Long\-horizon planning for assembly under semantic constraints has a combinatorially large search space, so we decompose the planning problem into the two subproblems\. In the goal selection phase, the agent selects the best set of recognised scene objects that can be used to assemble the goal conceptγ\\gammaby building a structure instantiating the assembly topologyGγG\_\{\\gamma\}\. Currently known constraints inSγS\_\{\\gamma\}are translated into ASP clauses \(see above\) and added to the ASP encoding of the goal selection problem, so that the agent avoids knowingly violating the constraints\.

Each possible goal structure is scored by the visual compatibility of part recognition, as measured by the sum of corresponding confidence scores, and the count of any violated constraints inSγS\_\{\\gamma\}\. The result of the goal selection process is that the agent makes decisions as to which objects are instances of which \(currently known\) atomic part concepts\. Our implementation uses the multi\-shot solving feature of the ASP solver Clingo\([4](https://arxiv.org/html/2608.13684#bib.bib4)\)for incremental optimisation of the score\. In case the agent fails to visually recognise the full set of objects needed for buildingγ\\gamma, our ASP encoding encourages partial planning towards fulfillingGγG\_\{\\gamma\}, after which the agent reports planning failure and requests feedback from the teacher \(see the “grounding failure” link in Fig\.\)\. Our encoding also accommodates re\-planning during execution after planning failures or teacher corrections\.

The output of the goal selection phase is passed to the join sequence planning phase, in which the agent plans a collision\-free order of joins\. Clingo’s capability to solve ASP programs modulo theories\([3](https://arxiv.org/html/2608.13684#bib.bib3)\)allows us to check whether there exists a collision\-free joining path between two subassemblies by integration of any arbitrary motion planner\. Any infeasible joins are discarded and rememberedduringplan search\. We record in App\.[G](https://arxiv.org/html/2608.13684#A7)unique motion\-planner calls as an auxiliary metric, since expensive collision checks can affect performance\. The planned action sequence is finally passed to the actuation module for execution\.

## 4Interactive Learning Procedures

### 4\.1Agent\-Teacher Interaction Strategies

We compare four interaction strategies that share the same architecture, task abstraction, visual backbone, planner, and oracle execution assumptions, but differ in what kinds of teacher feedback they can use to update memory\.

NoLabels\+CaseMemory:The agent does not share a NL vocabulary for the part concepts inΠ\\Pi\. It cannot interpret concept labels or generic statements, but it can still store correction cases over its own internal concept symbols when teacher interventions indicate that one attempted choice should be replaced by another\. This serves as a diagnostic lower bound for learning without shared part labels\.

LabelsOnly:The agent can use shared vocabularies obtained from NL dialogue for part concept labelling, but does not store semantic constraints from teacher corrections\.

Labels\+CaseMemory:The agent can use shared part concept labels and also stores non\-generic correction cases of the form “Uset​h​i​s1this\_\{1\}instead oft​h​i​s2this\_\{2\}”, but cannot interpret generic statements that encode constraints inSγS\_\{\\gamma\}\.

Labels\+TeacherRules:This is our full approach\. The agent can use concept labels and teacher\-provided generic statements that are translated into symbolic constraints\.

### 4\.2Learning Visuals and Geometries of Parts

The three label\-aware strategies acquire labelled exemplars for part concepts directly through dialogue and narrated demonstrations, enabling targeted updates to visual grounding\. By contrast, NoLabels\+CaseMemory lacks a shared part vocabulary and relies on indirect signals, including post\-hoc analysis of completed assemblies and teacher interruptions, which introduces additional noise\. The extraction of 3D geometries and join poses follows the same procedure for all strategies; label\-aware agents can immediately identify novel parts via linguistic cues, whereas NoLabels\+CaseMemory relies on recognition uncertainty\. The detailed procedures for visual grounding and 3D geometry extraction are provided in App\.[D](https://arxiv.org/html/2608.13684#A4)\.

### 4\.3Learning Semantic Constraints

Labels\+TeacherRules learns semantic constraints directly from generic teacher statements, as shown in Fig\.\. When the agent violates a semantic constraint, the teacher states the violated rule, and the agent translates it into FOL and ASP clauses added toSγS\_\{\\gamma\}\. LabelsOnly performs no such semantic\-memory update\.

The \+CaseMemory strategies store non\-generic correction memories derived from teacher interventions\. When a violation is signalled by “Uset​h​i​s1this\_\{1\}instead oft​h​i​s2this\_\{2\}”, the agent stores a most\-specific correction case that discourages usingt​h​i​s2this\_\{2\}\(as recognised\) when an unusedt​h​i​s1this\_\{1\}\(as recognised\) is available\. For example, suppose the agent is tasked to build a dump truck and is notified to use a red hemtt cabin instead of a blue quad cabin\. The agent would then add a new ASP constraint as follows; informally, ‘when building a dump truck, do not use a blue quad cabin when a red hemtt cabin is available’:

v​i​o​l​a​t​e​d​\(ci\)←d​u​m​p​T​r​u​c​k​\(X\),q​u​a​d​C​a​b​i​n​\(Y\),b​l​u​e​\(Y\),h​a​v​e​\(X,Y\),violated\(c\_\{i\}\)\\leftarrow dumpTruck\(X\),quadCabin\(Y\),blue\(Y\),have\(X,Y\),

h​e​m​t​t​C​a​b​i​n​\(Z\),r​e​d​\(Z\),not​h​a​v​e​\(X,Z\)\.\\hskip 72\.26999pthemttCabin\(Z\),red\(Z\),\\text\{ not \}have\(X,Z\)\.

This case\-memory mechanism does not abstract over corrections or discard irrelevant literals; inducing general constraints from cases, e\.g\. via ILP or anti\-unification, is a natural direction for future work\.

## 5Experiments

### 5\.1Experimental Design and Evaluation

We test the interaction strategies on 30 datasets from the simulated toy\-truck domain described in App\.[E](https://arxiv.org/html/2608.13684#A5), each comprising 40 randomly sampled online assembly episodes\. The domain consists of 22 atomic part types inΠ\\Piand 6 truck types inΣ\\Sigma\. The first 5 problems in each dataset are a ‘warm\-up’, where the agent is tasked to assemble a generic truck without any distractors \(D=∅D=\\varnothing\)\. After this, unforeseen subtypes of trucks and parts are constantly introduced along with randomly sampled distractors \(D≠∅D\\neq\\varnothing\)\. There are no distinct ‘training’ and ‘testing’ splits—learning happensduringtask execution and the interaction with the teacher\. The primary comparison is among the three label\-aware strategies; NoLabels\+CaseMemory is retained as a diagnostic lower bound for learning without shared part labels\.

We use cumulative regret as the primary evaluation metric: i\.e\., the accumulated counts of errors made across a sequence of 40 planning problems\. We track five types of errors in total: 1\) joining structurally incompatible pairs; 2\) joining at incorrect pose; 3\) using a distractor part; 4\) failures to ground a needed part \(addressed by “Is there a X?” questions\); and 5\) ASP planner timeouts due to excessive grounding uncertainty\. These five error types are weighted uniformly\. We also monitor how the agents’ visual grounding evolves over time, measured as mean F1 score curves obtained across each dataset\. We report the cumulative regret and mean F1 curves averaged over the 30 datasets with 95% confidence intervals\.

### 5\.2Results and Discussion

\\subfigure\[Cumulative regret curves\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/cumulative_regret.png)\\subfigure\[Mean F1 curves\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/mean_f1.png)

Fig\.[5\.2](https://arxiv.org/html/2608.13684#S5.SS2)presents the averaged cumulative regret curves across 40 episodes for each interaction strategy\. NoLabels\+CaseMemory incurs substantially higher regret than the label\-aware strategies, reflecting the difficulty of learning novel part concepts without a shared vocabulary\. Among the label\-aware strategies, LabelsOnly and Labels\+CaseMemory exhibit very similar regret curves, with overlapping confidence intervals and no reliable advantage from storing most\-specific correction cases\. This suggests that simple case memory is not an effective substitute for reusable semantic constraints in this domain\. By contrast, Labels\+TeacherRules consistently achieves lower cumulative regret, showing that teacher\-provided generic constraints improve online assembly learning beyond part labels and non\-generic correction memories\. App\.[F](https://arxiv.org/html/2608.13684#A6)provides a detailed breakdown of the five error types for all interaction strategies\.

Fig\.[5\.2](https://arxiv.org/html/2608.13684#S5.SS2)shows the evolution of mean F1 scores for visual grounding\. The three label\-aware strategies obtain comparable grounding performance, whereas NoLabels\+CaseMemory lags behind because novel part types are introduced without explicit labelling signals\. Crucially, the lower regret of Labels\+TeacherRules is not explained by better visual grounding: its mean F1 is comparable to LabelsOnly and Labels\+CaseMemory\. The gain therefore comes from how generic constraints are used by the symbolic planner to avoid semantically invalid assemblies, not merely from improved part classification\. Together, these results isolate the value of generic teacher statements as a direct channel for communicating reusable symbolic constraints during deployment\.

## 6Related Work

Our work relates to concept\-based and explanatory interactive learning, where concept\-level corrections, prototype interactions, and self\-explanations revise neural or neurosymbolic models\([19](https://arxiv.org/html/2608.13684#bib.bib17);[18](https://arxiv.org/html/2608.13684#bib.bib18);[17](https://arxiv.org/html/2608.13684#bib.bib19)\)\. These approaches demonstrate the value of concept\-level feedback, but typically focus on revising learned models with a largely fixed task vocabulary\. By contrast, our agent overcomes its unawareness of part predicates, subassembly concepts, topology, and semantic constraints during deployment, with the objective of immediate replanning rather than representation\-level model debugging\.

Recent VLM\-based neurosymbolic systems map perception and language into symbolic structures such as predicates, visual programs, or planning constraints\([1](https://arxiv.org/html/2608.13684#bib.bib1);[21](https://arxiv.org/html/2608.13684#bib.bib21);[8](https://arxiv.org/html/2608.13684#bib.bib8)\)\. Human\-guided systems such as LARA likewise integrate communication, learning, reasoning, and planning\([7](https://arxiv.org/html/2608.13684#bib.bib7)\)\. Our work studies a complementary regime: a human teacher communicates generic semantic constraints as verified symbolic knowledge during task execution, and an ASP planner immediately uses them for object\-role assignment and join planning\. Closest to our own work are[14](https://arxiv.org/html/2608.13684#bib.bib14);[15](https://arxiv.org/html/2608.13684#bib.bib15), which use generic language for visually grounded concept and domain\-theory learning; we extend this line to long\-horizon assembly planning under quantified semantic constraints\.

## 7Conclusion and Future Directions

We presented a neurosymbolic agent that learns semantic assembly tasks through controlled natural language interaction and demonstrations\. The study focuses on deployment\-time adaptation: the agent begins without the relevant part concepts, subassembly concepts, topology, or semantic constraints, and updates visual exemplars, symbolic memory, and \(re\)plans during task execution\. Our empirical findings show that teacher\-provided generic constraints improve online assembly learning in this benchmark beyond part labels and non\-generic correction memories alone, with gains arising from symbolic planning rather than visual grounding alone\. Together, these results illustrate a form of neurosymbolic adaptation in which explicit symbolic knowledge acquired during interaction immediately reshapes action selection over uncertain perceptual groundings\. The study remains limited to a controlled simulated domain and a modular, primarily perception\-to\-reasoning form of neurosymbolic coupling\. Future work should address more robust perception and language interfaces, cross\-domain assembly benchmarks, abstraction from correction cases, and tighter symbolic\-to\-neural feedback where representation\-level revision is needed\.

###### acknowledgments\-disclosure\-of\-funding\.

This work was supported by Informatics Global PhD Scholarships, funded by the School of Informatics at The University of Edinburgh\. Ramamoorthy is supported by a UKRI Turing AI World Leading Researcher Fellowship on AI for Person\-Centred and Teachable Autonomy \(grant EP/Z534833/1\)\. We thank the anonymous reviewers for their feedback on an earlier draft of this paper, and Rimvydas Rubavicius and Gautier Dagan for continued feedback over the course of this research\. Naver Labs Europe, the current employer of the attending author, sponsored registration and travel for in\-person attendance to the conference venue\.

## References

- Athalyeet al\.\(2026\)A\. Athalye, N\. Kumar, T\. Silver, Y\. Liang, J\. Wang, T\. Lozano\-Pérez, and L\. P\. KaelblingFrom pixels to predicates: learning symbolic world models via pretrained vlms\.IEEE Robotics and Automation Letters\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p2.1)\.
- Copestake and Flickinger \(2000\)A\. A\. Copestake and D\. FlickingerAn open source grammar development environment and broad\-coverage english grammar using hpsg\.\.InLREC,pp\. 591–600\.Cited by:[§3\.2](https://arxiv.org/html/2608.13684#S3.SS2.p1.1)\.
- Gebseret al\.\(2016\)M\. Gebser, R\. Kaminski, B\. Kaufmann, M\. Ostrowski, T\. Schaub, and P\. WankoTheory solving made easy with clingo 5\.InTechnical Communications of the 32nd International Conference on Logic Programming \(ICLP 2016\),pp\. 2–1\.Cited by:[§3\.5](https://arxiv.org/html/2608.13684#S3.SS5.p3.1)\.
- Gebseret al\.\(2019\)M\. Gebser, R\. Kaminski, B\. Kaufmann, and T\. SchaubMulti\-shot asp solving with clingo\.Theory and Practice of Logic Programming19\(1\),pp\. 27–82\.Cited by:[§3\.5](https://arxiv.org/html/2608.13684#S3.SS5.p2.1)\.
- Jianget al\.\(2022\)Y\. Jiang, Z\. Huang, B\. Yang, and W\. YangA review of robotic assembly strategies for the full operation procedure: planning, execution and evaluation\.Robotics and Computer\-Integrated Manufacturing78,pp\. 102366\.Cited by:[§1](https://arxiv.org/html/2608.13684#S1.p1.1)\.
- Kirillovet al\.\(2023\)A\. Kirillov, E\. Mintun, N\. Ravi, H\. Mao, C\. Rolland, L\. Gustafson, T\. Xiao, S\. Whitehead, A\. C\. Berg, W\. Lo,et al\.Segment anything\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 4015–4026\.Cited by:[Appendix C](https://arxiv.org/html/2608.13684#A3.p2.1)\.
- Kokelet al\.\(2022\)H\. Kokel, M\. Das, R\. Islam, J\. Bonn, J\. Cai, S\. Dan, N\. Anjali, P\. Jayannavar, D\. J\. Rao, J\. Hockenmaier, S\. Natarajan, M\. Palmer, and D\. RothLARA – human\-guided collaborative problem solver: effective integration of learning, reasoning and communication\.The Tenth Annual Conference on Advances in Cognitive Systems \(ACS\)\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p2.1)\.
- Kumaret al\.\(2026\)N\. Kumar, W\. Shen, F\. Ramos, D\. Fox, T\. Lozano\-Pérez, L\. P\. Kaelbling, and C\. R\. GarrettOpen\-world task and motion planning via vision\-language model generated constraints\.IEEE Robotics and Automation Letters\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p2.1)\.
- Leeet al\.\(2024\)R\. K\. Lee, H\. Zheng, and Y\. LuHuman\-robot shared assembly taxonomy: a step toward seamless human\-robot knowledge transfer\.Robotics and Computer\-Integrated Manufacturing86,pp\. 102686\.Cited by:[§1](https://arxiv.org/html/2608.13684#S1.p1.1)\.
- Lifschitz \(2019\)V\. LifschitzAnswer set programming\.Vol\.3,Springer Cham\.Cited by:[§2\.3](https://arxiv.org/html/2608.13684#S2.SS3.p1.1)\.
- Liuet al\.\(2022\)Y\. Liu, Y\. Wen, S\. Peng, C\. Lin, X\. Long, T\. Komura, and W\. WangGen6d: generalizable model\-free 6\-dof object pose estimation from rgb images\.InEuropean Conference on Computer Vision,pp\. 298–315\.Cited by:[Appendix C](https://arxiv.org/html/2608.13684#A3.p5.1)\.
- Oquabet al\.\(2023\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. HAZIZA, F\. Massa, A\. El\-Nouby,et al\.DINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research\.Cited by:[Appendix C](https://arxiv.org/html/2608.13684#A3.p3.1)\.
- Örneket al\.\(2024\)E\. P\. Örnek, Y\. Labbé, B\. Tekin, L\. Ma, C\. Keskin, C\. Forster, and T\. HodanFoundpose: unseen object pose estimation with foundation features\.InEuropean Conference on Computer Vision,pp\. 163–182\.Cited by:[Appendix C](https://arxiv.org/html/2608.13684#A3.p5.1)\.
- Parket al\.\(2023\)J\. Park, A\. Lascarides, and S\. RamamoorthyInteractive acquisition of fine\-grained visual concepts by exploiting semantics of generic characterizations in discourse\.InProceedings of the 15th International Conference on Computational Semantics,pp\. 318–331\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p2.1)\.
- Parket al\.\(2025\)J\. Park, A\. Lascarides, and S\. RamamoorthyLearning visually grounded domain ontologies via embodied conversation and explanation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 14361–14368\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p2.1)\.
- Schonberger and Frahm \(2016\)J\. L\. Schonberger and J\. FrahmStructure\-from\-motion revisited\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 4104–4113\.Cited by:[Appendix C](https://arxiv.org/html/2608.13684#A3.p4.1)\.
- Stammeret al\.\(2024\)W\. Stammer, F\. Friedrich, D\. Steinmann, M\. Brack, H\. Shindo, and K\. KerstingLearning by self\-explaining\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=bpjU7rLjJ7)Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p1.1)\.
- Stammeret al\.\(2022\)W\. Stammer, M\. Memmel, P\. Schramowski, and K\. KerstingInteractive disentanglement: learning concepts by interacting with their prototype representations\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 10317–10328\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p1.1)\.
- Stammeret al\.\(2021\)W\. Stammer, P\. Schramowski, and K\. KerstingRight for the right concept: revising neuro\-symbolic concepts by interacting with their explanations\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 3619–3629\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p1.1)\.
- Tranet al\.\(2023\)S\. C\. Tran, E\. Pontelli, M\. Balduccini, and T\. SchaubAnswer set planning: a survey\.Theory and Practice of Logic Programming23\(1\),pp\. 226–298\.Cited by:[§2\.3](https://arxiv.org/html/2608.13684#S2.SS3.p1.1)\.
- Wüstet al\.\(2026\)A\. Wüst, W\. Stammer, H\. Shindo, L\. Helff, D\. S\. Dhami, and K\. KerstingSynthesizing visual concepts as vision\-language programs\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 17346–17356\.Cited by:[§6](https://arxiv.org/html/2608.13684#S6.p2.1)\.
- Zhu and Hu \(2018\)Z\. Zhu and H\. HuRobot learning from demonstration in robotic assembly: a survey\.Robotics7\(2\),pp\. 17\.Cited by:[§1](https://arxiv.org/html/2608.13684#S1.p1.1)\.

## Appendix AASP Encodings of Planning Subproblems

The ASP program given in Fig\.,andencodes the first planning subproblem described in the paper, i\.e\., goal selection\. Note that this ASP encoding assumes a solver that supports multi\-shot solving feature like Clingo\.

% Make choice for the top node \(0\), which represents the build target subassembly1\{ node\_sa\_template\(0,S,I\) : possible\_root\_template\(S,I\) \}1\.% Determine possible template options for the root node, from the specified% build target type and available supertype\-subtype relationspossible\_root\_template\(S,I\) :\- build\_target\(S\), template\_option\(S,I\)\.possible\_root\_template\(S2,I\) :\-build\_target\(S1\), subtype\_of\(S1,S2\), template\_option\(S2,I\)\.% Supertype\-subtype relation is transitivesubtype\_of\(T1,T3\) :\- subtype\_of\(T1,T2\), subtype\_of\(T2,T3\)\.% Annotating each child of a subassembly node N with its required atomic part% concept typenode\_atomic\(n\(N,NS\),P\) :\- node\_sa\_template\(N,S,I\), req\_atomic\(S,I,NS,P\)\.% Meanwhile, a subassembly template may be directly specified for a nodenode\_sa\_template\(n\(N,NS\),S2,I2\) :\-node\_sa\_template\(N,S1,I1\), req\_template\(S1,I1,NS,S2,I2\)\.% Project to specify subassembly type of subassembly nodesnode\_sa\(N,S\) :\- node\_sa\_template\(N,S,I\)\.node\_sa\(0,S\) :\- build\_target\(S\)\. % Also account for build target subtype% Supertype\-subtype info applied for substructuresnode\_sa\(N,S2\) :\- node\_sa\(N,S1\), subtype\_of\(S1,S2\)\.% Ancestor\-descendant relations among nodescomponent\_node\(n\(N,NS\),N\) :\- node\_atomic\(n\(N,NS\),\_\)\.component\_node\(n\(N,NS\),N\) :\- node\_sa\(n\(N,NS\),\_\)\.component\_node\(N1,N3\) :\- component\_node\(N1,N2\), component\_node\(N2,N3\)\. % Transitive% An object that is likely to be an instance of a type may fill an atomic node% with matching type\. Each object can fill up to one atomic node, and each% atomic node can be filled by up to one object\.can\_fill\(O,N,P\) :\- node\_atomic\(N,P\), type\_likely\(O,P,PR\)\.can\_fill\(O,N,P1\) :\- node\_atomic\(N,P2\), subtype\_of\(P1,P2\), type\_likely\(O,P1,PR\)\.\{ fill\_node\(O,N\) \} :\- can\_fill\(O,N,\_\), not must\_unify\(\_,N\)\.:\- type\_likely\(O,\_,\_\), \#count \{ N : fill\_node\(O,N\) \} \> 1\.:\- node\_atomic\(N,\_\), \#count \{ O : fill\_node\(O,N\) \} \> 1\.% Once determined an object to fill a specific atomic node, select exactly one% part \(sub\)type to commit to among the visually licensed options1\{ type\_committed\(O,P\) : can\_fill\(O,N,P\) \}1 :\- fill\_node\(O,N\), can\_fill\(O,N,\_\)\.% Supertype\-subtype info percolates upwardstype\_committed\(O,P2\) :\- type\_committed\(O,P1\), subtype\_of\(P1,P2\)\.% Always use objects labeled by user in response to agent’s "Is there a ~?" queries:\- certified\_label\(O\), not fill\_node\(O,\_\)\.

% Identifying all objects selected to fill all descendant atomic nodes of a% subassembly nodecomponent\_obj\(O,N\) :\- node\_sa\(N,S\), component\_node\(ND,N\), fill\_node\(O,ND\)\.% For tracking which nodes will have to be connected at which contact pointsto\_connect\(D1,D2,CP1,CP2\) :\-node\_sa\_template\(N,S,I\),fits\_signature\(D1,N,NS1,SG1\), fits\_signature\(D2,N,NS2,SG2\),connection\_signature\(S,I,NS1,NS2,SG1,SG2,CP1,CP2\)\.fits\_signature\(n\(N,NS\),N,NS,NS\) :\- node\_atomic\(n\(N,NS\),\_\)\.fits\_signature\(D,N,NS,c\(NS,SG\)\) :\-fits\_signature\(D,n\(N,NS\),\_,SG\), node\_sa\_template\(n\(N,NS\),\_,\_\)\.% Handling additional constraints due to already assembled parts\. Atomic% part type of existing object may or may not be specified, represented% by arity of ext\_obj predicate \(ext\_obj/2 vs\. ext\_obj/1\)\. Similarly,% contact sites between two joined existing parts may or may not be known% \(ext\_conn/4 vs\. ext\_conn/2\)\.fresh\_obj\(O\) :\- component\_obj\(O,\_\), not ext\_obj\(O\), not ext\_obj\(O,\_\)\.% ’Fresh’ if not included in some subassembly% If an existing object with known part type has exactly one atomic node with% the matching type, unifymust\_unify\(O,N\) :\- ext\_obj\(O,P\), 1\{ node\_atomic\(\_,P\) \}1, node\_atomic\(N,P\)\.% If a neighbor of a uniquely unified object connects with another by known% contact site, unify \(assumption here is that contact sites of each atomic part% are all uniquely distinguishable\)must\_unify\(O1,N1\) :\- must\_unify\(O2,N2\), ext\_conn\(O1,O2,CP1,CP2\),to\_connect\(N1,N2,CP1,CP2\)\.% If a neighbor of a uniquely unified object has known type, and there exist% exactly one node with matching type in the neighborhood of the uniquely unified% node, unifymust\_unify\(O1,N1\) :\- must\_unify\(O2,N2\), ext\_obj\(O1,P1\), ext\_conn\(O1,O2\),1\{ node\_atomic\(N,P1\) : to\_connect\(N,N2,\_,\_\) \}1,node\_atomic\(N1,P1\), to\_connect\(N1,N2,\_,\_\)\.% If a neighbor of a uniquely unified object has known type, and there exist% more than one nodes with matching type in the neighborhood of the uniquely% unified node, \*MAY\* unify with one of themmay\_unify\(O1,N1\) :\- must\_unify\(O2,N2\), ext\_obj\(O1,P1\), ext\_conn\(O1,O2\),2\{ node\_atomic\(N,P1\) : to\_connect\(N,N2,\_,\_\) \},node\_atomic\(N1,P1\), to\_connect\(N1,N2,\_,\_\)\.% If one side of existing object connection uniquely unifies to an atomic node% while the other’s type is not specified, the latter \*MAY\* be unified with one% of other atomic nodes connected to the unified nodemay\_unify\(O1,N1\) :\- must\_unify\(O2,N2\), ext\_obj\(O1\), ext\_conn\(O1,O2\),to\_connect\(N1,N2,\_,\_\)\.

% Observe must\_unify relationsfill\_node\(O,N\) :\- must\_unify\(O,N\)\.% Different objects cannot be forced to unify with the same node:\- must\_unify\(O1,N1\), must\_unify\(O2,N2\), O1 \!= O2, N1 = N2\.% Observe ext\_conn relations:\- ext\_conn\(O1,O2,\_,\_\), fill\_node\(O1,N1\), fill\_node\(O2,N2\), not to\_connect\(N1,N2,\_,\_\)\.:\- ext\_conn\(O1,O2\), fill\_node\(O1,N1\), fill\_node\(O2,N2\), not to\_connect\(N1,N2,\_,\_\)\.% Supertype\-subtype relations for ext\_objsext\_obj\(O,P2\) :\- ext\_obj\(O,P1\), subtype\_of\(P1,P2\)\.% Edge symmetricityto\_connect\(N2,N1,CP2,CP1\) :\- to\_connect\(N1,N2,CP1,CP2\)\.ext\_conn\(O2,O1,CP2,CP1\) :\- ext\_conn\(O1,O2,CP1,CP2\)\.ext\_conn\(O2,O1\) :\- ext\_conn\(O1,O2\)\.% Compute average part compatibility score by taking average across all atomic% nodes filled by recognized objectsnode\_score\(N,PR\) :\-node\_atomic\(N,\_\), fill\_node\(O,N\), type\_committed\(O,P\), type\_likely\(O,P,PR\)\.avg\_score\(TS/NN\) :\- TS = \#sum \{ NS,N : node\_score\(N,NS\) \},NN = \#count \{ N : node\_atomic\(N,\_\) \}, NN \!= 0\.% Prevent any violation of universally quantified constraints\. Implemented as% hard constraint, as violation can be evaded by deciding to not fill corresponding% nodes with recognized objects\.:\- forall\_violation\(\_\)\.% Penalize any violation of existentially quantified constraints\. Implemented% as soft constraint, as plans violating these can still be admitted, where% absence of required of parts will be handled by queries to user\. In contrast,% if implemented as hard constraints, many sane partial plans will be eliminated\.total\_penalty\(TP\) :\- TP = \#sum \{ 30,EC : exists\_violation\(EC\) \}\.% Final goal configuration scorefinal\_score\(AS\-TP\) :\- avg\_score\(AS\), total\_penalty\(TP\)\.\#program check\(c\)\.\#external query\(c\)\.:\- final\_score\(FS\), query\(c\), FS <= c\.

The ASP program given in Fig\.andencodes the second planning subproblem, i\.e\., join sequence planning\. Note that this ASP encoding also assumes a solver that supports multi\-shot solving feature, in addition to ASP\-modulo\-theory solving feature, again like Clingo\.

%% Rules common to answer set planning\#program base\.holds\(F,0\) :\- init\(F\)\.\#program step\(t\)\.holds\(F,t\+1\) :\- holds\(F,t\), not \-holds\(F,t\+1\)\.\-holds\(F,t\+1\) :\- \-holds\(F,t\), not holds\(F,t\+1\)\.1\{ occ\(A,t\) : possible\(A,t\) \}1\.\#program check\(t\)\.\#external query\(t\)\.:\- query\(t\), not goal\(t\)\.%% Rules common to \(our way of encoding\) assembly domains \- Static laws\#program base\.% Part connection is a symmetric relationto\_connect\(O2,O1\) :\- to\_connect\(O1,O2\)\.% Track every occasion where a subassembly formed in the previous timestep% doesn’t get used in the immediate next timestep \(identifiable by index\);% total count of such occurrences will be minimized\#program step\(t\)\.penalize\(t\) :\-occ\(join\(S1,O1,S2,O2\),t\), holds\(max\_sa\_index\(S\),t\), S1 \!= S, S2 \!= S\.%% Rules common to assembly domains \- Dynamic laws\#program step\(t\)\.% join/4: Assembling two subassemblies at specified object; abstracts% aligning & tightening\.% \(Note: This implementation assumes we always assemble from right to left,% without loss of generality\)possible\(join\(S1,O1,S2,O2\),t\) :\-S1 \!= S2, O1 \!= O2,holds\(part\_of\(O1,S1\),t\), holds\(part\_of\(O2,S2\),t\),to\_connect\(O1,O2\), not holds\(connected\(O1,O2\),t\)\.holds\(connected\(O1,O2\),t\+1\) :\- occ\(join\(S1,O1,S2,O2\),t\)\.holds\(connected\(O2,O1\),t\+1\) :\- occ\(join\(S1,O1,S2,O2\),t\)\. % Symmetricholds\(max\_sa\_index\(S\+1\),t\+1\) :\- occ\(join\(S1,O1,S2,O2\),t\), holds\(max\_sa\_index\(S\),t\)\.\-holds\(max\_sa\_index\(S\),t\+1\) :\- occ\(join\(S1,O1,S2,O2\),t\), holds\(max\_sa\_index\(S\),t\)\.\-holds\(part\_of\(O,S1\),t\+1\) :\- occ\(join\(S1,\_,S2,\_\),t\), holds\(part\_of\(O,S1\),t\)\.\-holds\(part\_of\(O,S2\),t\+1\) :\- occ\(join\(S1,\_,S2,\_\),t\), holds\(part\_of\(O,S2\),t\)\.holds\(part\_of\(O,S\+1\),t\+1\) :\-occ\(join\(S1,\_,S2,\_\),t\), holds\(max\_sa\_index\(S\),t\), holds\(part\_of\(O,S1\),t\)\.holds\(part\_of\(O,S\+1\),t\+1\) :\-occ\(join\(S1,\_,S2,\_\),t\), holds\(max\_sa\_index\(S\),t\), holds\(part\_of\(O,S2\),t\)\.

% Note: We don’t consider ’disassemble’ operation here; would need to treat% subassembly predicate as fluent if we were to do this% Minimize number of times occasions where agent does not immediately% re\-use the subassembly just assembled in the previous timestep for the% next join action, thus increasing the number of primitive actions needed\.% It is important that this optimization statement is placed here, as part% of the program fragment step\(t\)\.\#minimize \{ 1,T : penalize\(T\) \}\.%% Goal condition; all object pairs to be connected have been ’connected’% at time step t, either truly or speculatively\#program check\(t\)\.goal\(t\) :\- holds\(connected\(O1,O2\),t\) : to\_connect\(O1,O2\)\.%% Problem init; common to all problems\#program base\.% Initial configsinit\(max\_sa\_index\(\-1\)\)\.

## Appendix BDetailed Account of Agent\-Teacher Interaction Flow

This section expands the compact description of the controlled interaction protocol in §[2\.4](https://arxiv.org/html/2608.13684#S2.SS4)\. Fig\.depicts the full set of teacher\-learner interaction branches used in the experiments\.

![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/interaction_flow.png)Each task begins with the teacher specifying the assembly goal\. If the agent is unaware of the assembly target, it reports its ignorance and the teacher responds either through a full demonstration or through a verbal definition, depending on context\. Demonstrations \(accompanied with NL narrations\) are preferred when new geometric relations or join poses must be learned\. Verbal definitions suffice if the agent already possesses the relevant intermediary concepts \(e\.g\., “A dump truck is a truck with a dumper”\)\.

If the agent believes it has sufficient knowledge, it generates a plan and begins execution\. During planning, the agent may fail to visually recognise a required atomic part \(which is guaranteed to exist inCC\)\. In such cases, the agent requests clarification and the teacher responds by pointing to the correct object\.

If the agent makes an error during execution, the teacher interrupts and asks the agent to explain its intended action\. The agent’s response reveals its current beliefs about the involved objects, and exposes two possible knowledge gaps\. First, the agent may have made a grounding error\. The teacher corrects this by negating the incorrect grounding and asserting the correct type\. Second, the grounding may be correct but the attempted action may violate a semantic constraint\. The teacher then points to a valid object and expresses the violated constraint as a generic NL statement\. After corrective feedback, the agent updates its knowledge and replans\.

## Appendix CVision Processing Module: Implementation details

The vision processing module parses the agent’s visual inputs from its RGB sensor into structured representations that other modules can further process\. The module deals with three visual subtasks: object detection \(including predictions of its types\); point cloud extraction; and pose estimation\.

The goal of object detection is to localise and to classify scene objects\. We use binary segmentation masks to represent localised object instances instead of bounding boxes since they allow more accurate visual feature extraction for classification\. Our primary focus is on learning to recognise classes of objects rather than their locations, since this directly interfaces with the goal of identifying novel parts\. While high\-quality, off\-the\-shelf segmentation models such as SAM\([6](https://arxiv.org/html/2608.13684#bib.bib6)\)could be used for object localization, it would introduce uncontrolled confounders for testing our main hypotheses\. As stated in §[2\.2](https://arxiv.org/html/2608.13684#S2.SS2), experiments use ground\-truth masks obtained from the simulated environment\.

We implement few\-shot open\-set classification by keeping a collection of lightweight binary classifiers for each atomic part concept that the learner is currently aware of\. \(This subset of ‘aware’ part conceptsΠ\\Pikeeps expanding as interaction with the teacher proceeds\.\) The binary classifiers are induced from positive and negative exemplars stored in the agent’s long\-term memory and gained from teacher interactions \(see §[2\.4](https://arxiv.org/html/2608.13684#S2.SS4)\)\. Our implementation uses the DINOv2\-base model\([12](https://arxiv.org/html/2608.13684#bib.bib12)\)as a feature extractor backbone, which takes a scene image and object masks as input, and outputs are fed into RBF\-kernel SVMs to yield per\-concept predictions\.

Atomic part concepts need to have proper representations of their 3D structures for accurate manipulation during assembly\. Under the type\-level geometry assumption stated in §[2\.2](https://arxiv.org/html/2608.13684#S2.SS2), each atomic part concept is represented by a point cloud\. When the learner encounters an atomic part type for the first time, and notices that it does not correspond to any previously known types, the learner registers a new atomic part concept inΠ\\Piand associates it with a point cloud representation extracted from the instance\. Given that visual inputs are single\-view RGB images, the learner picks up the novel part concept instance and collects a set of images of the object observed from different viewpoints with known poses\. The feature extractor processes each image to obtain per\-pixel embeddings at a lower resolution, which can be used as point descriptors to be fed into a photogrammetric point cloud extraction algorithm\. Our implementation employs COLMAP\([16](https://arxiv.org/html/2608.13684#bib.bib16)\)for point cloud reconstruction\.

Estimation of 6D poses of objects is necessary to achieve accurate joining of parts at specified relative poses\. Few\-shot or zero\-shot object pose estimation with RGB images has witnessed significant research progress with the advent of powerful vision foundation models\([11](https://arxiv.org/html/2608.13684#bib.bib11);[13](https://arxiv.org/html/2608.13684#bib.bib13)\)\. As stated in §[2\.2](https://arxiv.org/html/2608.13684#S2.SS2), experiments use ground\-truth object poses from the simulated environment after each manipulation action\.

## Appendix DPer\-strategy Details on Learning from Visual Data

### D\.1Learning Visual Grounding of Parts

The agent’s visual grounding performance is determined by the binary classifiers induced from the positive and negative exemplar setsχ\+/−π\\chi^\{\+/\-\}\_\{\\pi\}stored in the visual XB\. The progress of learning to visually recognise atomic partsΠ\\Pilargely depends on acquiring labelled concept exemplars through interactions with the teacher\. Consequently, the visual grounding learning process differs between NoLabels\+CaseMemory and the label\-aware strategies, since only the latter have access to NL labels for concepts inΠ\\Pi\.

Label\-aware agents directly obtain exemplar labelling through concept labelling statements like “This is adumper”, “This is not aquad\_cabin”, “Here is alarge\_wheel”\. As illustrated in Fig\., these labelling statements are given after the agent’s utterance that reveals how it has mistaken a part type for another \(“I meant to join this X and this Y\.”\) or failed to ground a needed instance \(“Is there a X on the table?”\)\. Concept labels are also provided when the teacher provides full demonstrations upon the agent’s ignorance of goal subassembly concepts, in which each action step is narrated in natural language\. These interactions are possible only when the agent shares a NL vocabulary with the teacher\.

In contrast, NoLabels\+CaseMemory agents are neither able to understand the teacher’s concept labelling statements nor to generate utterances that expose its imperfect grounding capability due to lack of a shared vocabulary\. In case of grounding failures, the agent cannot ask “Is there a X on the table?” and has to report “I cannot find a part I need on the table” instead, after which the teacher will simply demonstrate a valid part join\. After agent mistakes, the teacher cannot ask the probing question “What were you trying to join?” because agent cannot answer with its current part recognitions, so the agent is simply interrupted; afterwards, the agent will immediately request a demonstration of a valid part join\.

Accordingly, the NoLabels\+CaseMemory baseline in our experiments has to rely on indirect learning signals to update their exemplar sets: post\-hoc analysis of each completed episode, and the teacher’s interruptions upon grounding mistakes\. Post\-hoc analysis of an episode updates positive exemplar setsχπ\+\\chi\_\{\\pi\}^\{\+\}by matching the topology of the completed structure againstGγG\_\{\\gamma\}via graph matching\. Negative exemplars can be inferred from teacher interruptions, which can serve as ‘pairwise negative labels’ of the involved object pairs\. Specifically, if the join of objectso1o\_\{1\}ando2o\_\{2\}are interrupted as invalid by the teacher due to incorrect grounding, and the agent had recognisedo1,o2o\_\{1\},o\_\{2\}asπ1,π2∈Π\\pi\_\{1\},\\pi\_\{2\}\\in\\Pirespectively, then it is not the case thato1o\_\{1\}is aπ1\\pi\_\{1\}ando2o\_\{2\}is aπ2\\pi\_\{2\}at the same time; otherwise, the join would not have been interrupted\. From the interruption, if the agent is certain thato1o\_\{1\}is indeed an instance ofπ1\\pi\_\{1\}, it can logically entailo2o\_\{2\}is not an instance ofπ2\\pi\_\{2\}, thereby updatingχπ2−\\chi\_\{\\pi\_\{2\}\}^\{\-\}\. Note that both label acquisition processes are susceptible to noise\.

### D\.2Learning 3D Geometries of Novel Parts and Joins

Across the four interaction strategies, the process itself remains the same for learning point cloud representations of atomic parts and joining poses\. The practical difference between NoLabels\+CaseMemory vs\. label\-aware agents in this regard has to do withwhenthe point cloud extraction procedure is triggered during interactions\. As explained in §[3\.1](https://arxiv.org/html/2608.13684#S3.SS1), the point cloud for an atomic part type is extracted when the agent encounters its instance for the first time in its operation\. In other words, the agent must first become aware that it has encountered a novel part type that is not inΠ\\Piprior to extracting its 3D geometry\. This is straightforward for label\-aware agents, who can immediately recognise when the teacher introduces a novel part type by use of neologisms\.

Conversely, NoLabels\+CaseMemory agents have to rely on their own judgements to determine whether an object is an instance of a novel atomic part type\. In our implementation, NoLabels\+CaseMemory agents register a novel part type when an object used in the teacher’s full demonstration obtains low likelihood scores from few\-shot visual recognition across all currently known types inΠ\\Pi\. Join poses involving the novel type are then obtained from the teacher’s demonstration\. Note that NoLabels\+CaseMemory agents may mistakenly identify a familiar part type as novel, or conversely, fail to recognise a novel part as unfamiliar, which will likely result in downstream errors during task execution\.

## Appendix EFull Description of Toy Truck Assembly Domain

Our simulated toy truck domain comprises 22 atomic part types inΠ\\Pi\. See Tab\.for the full list of the part types, whether they can have colour attributes, and example visuals\.

Π\\Pialso includes their supertypes:

- •\{cabin,loadchassis\_center,fender,wheel,fl\_fender,fr\_fender,bl\_fender,br\_fender,normal\_fender,large\_fender\}\.

See the domain theoryΩ\\Omegafor hyper/hyponymy relations that hold among them\.

The set of subassembly conceptsΣ\\Sigmainclude goal concepts, which are specified as task goals, and intermediary concepts, which are semantically meaningful substructures of goal concepts\.

- •Goal subassemblies: \{truck,fire\_truck,dump\_truck,container\_truck,missile\_truck\}\.
- •Non\-goal subeassemblies: \{truck\_front,truck\_back,fl\_fw\_unit,fr\_fw\_unit,bl\_fw\_unit,br\_fw\_unit,normal\_fl\_fw\_unit,normal\_fr\_fw\_unit,normal\_bl\_fw\_unit,normal\_br\_fw\_unit, large\_fl\_fw\_unit,large\_fr\_fw\_unit,large\_bl\_fw\_unit,large\_br\_fw\_unit\}\.

Likewise, see the domain theoryΩ\\Omegafor hyper/hyponymy relations that hold among them\. See the assembly topology graphsGγG\_\{\\gamma\}in Fig\.for the makeups of the subassembly concepts\.

The domain theoryΩ\\Omegafeatures hyper/hyponymy relations and holo/meronymy relations\. Fig\.depicts all hyper/hyponymy relations among concepts inΠ\\PiandΣ\\Sigma\. Holo/meronymy relations can be induced from the assembly topology graphsGγG\_\{\\gamma\}in Fig\.\.

The assembly topology graphsGγG\_\{\\gamma\}describe how each subassembly inΣ\\Sigmais composed from atomic parts inΠ\\Pior other subassemblies inΣ\\Sigma\.GγG\_\{\\gamma\}for the truck subtypesγ∈\\gamma\\in\{fire\_truck,dump\_truck,container\_truck,missile\_truck\} inherit fromGtruckG\_\{\\text\{\{truck\}\}\}; they are further defined with their definitional semantic constraints inSγS\_\{\\gamma\}\. Fig\.illustrates how the ‘fully flattened’ topology of atrucklooks like\.

Finally, Tab\.provides the full list of the semantic constraintsSγS\_\{\\gamma\}forγ∈Σ\\gamma\\in\\Sigma, merged into a single table\. Note how many of the constraints, or their more specific variants, cannot be obtained by the most\-specific correction memories used by the \+CaseMemory strategies\.

## Appendix FBreakdown of Cumulative Regrets

Fig\.provides a detailed breakdown of the averaged cumulative regret curves reported in the main paper\.

\\subfigure\[Structurally incompatible join attempt\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/plots/subtype_num_invalid_pickup.png)\\subfigure\[Join at incorrect pose\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/plots/subtype_num_invalid_join.png)\\subfigure\[Distractor usage\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/plots/subtype_num_distractor_pickup.png)\\subfigure\[Grounding failure\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/plots/subtype_num_search_failure.png)\\subfigure\[Planner timeout\]![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/plots/subtype_num_planning_forfeiture.png)

Note in particular how Labels\+TeacherRules agents make fewer distractor\-usage mistakes as learning proceeds\. The idiosyncratic behaviours of NoLabels\+CaseMemory curves, namely the seemingly increasing error counts for most types after the warm\-up problems, are due to the fact that they need full demonstrations for novel truck types, unlike label\-aware agents for which verbal definitions suffice; such demonstrations do not count towards any error type\. It is randomly determined at which point novel truck types are introduced in each dataset, so the numbers are averaged out across the datasets, giving the false sense of ‘increasing errors’ as the learning progresses\. Indeed, after sufficient number of episodes, NoLabels\+CaseMemory agents start to show learning progress again for most error types\.

## Appendix GAuxiliary Metric: Unique Motion Planner Calls

Fig\.provides the accumulated unique calls to the motion planner integrated into ASP planning procedure, invoked for collision\-free joining path checks\.

![[Uncaptioned image]](https://arxiv.org/html/2608.13684v1/figures/plots/subtype_num_collision_queries.png)NoLabels\+CaseMemory makes substantially more motion\-planner calls than the label\-aware strategies, largely because poorer grounding produces more invalid object choices and hence more replanning\. Among the label\-aware strategies, Labels\+TeacherRules makes fewer calls because semantic constraints are incorporated already in the goal\-selection ASP subproblem: object\-role assignments that would violate known constraints are rejected or penalised before the join\-sequence planner invokes collision checks\. This can reduce calls to the motion planner when semantically invalid distractor choices would otherwise remain geometrically feasible\. Since this metric is auxiliary and sensitive to implementation details of the planner and collision checker, we do not use it as primary evidence for the semantic\-learning claim\.