Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction

# Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction
Source: [https://arxiv.org/html/2607.08233](https://arxiv.org/html/2607.08233)
Antonia Wüst1Inga Ibs3,4Wasu Top Piriyakulkij5Wolfgang Stammer6Constantin Rothkopf2,3,4Kevin Ellis5Kristian Kersting1,2,4,71AIML Lab, TU Darmstadt;2Hessian Center for AI \(hessian\.AI\); 3Psychology of Information Processing, TU Darmstadt;4Centre for Cognitive Science, TU Darmstadt; 5Cornell University;6Max Planck Institute for Informatics, SIC;7German Center for AI \(DFKI\);

###### Abstract

A central challenge in building intelligent systems is enabling agents to jointly perceive complex inputs, form hypotheses about hidden patterns, and design informative experiments to test them\. To study this problem, we proposeZendoWorld, a controlled interactive environment in which agents must infer a logical rule about visual game observations, acquire information by proposing new scenes, and refine their hypotheses based on feedback from the game environment\. We evaluate several agents spanning pure VLM reasoning, Bayesian particle filtering, dynamic concept discovery, and neuro‑symbolic methods\. Our main findings are: \(1\) high accuracy in predicting labels for observed examples does not imply recovery of the underlying rule; \(2\) perception and induction are distinct bottlenecks for different agent classes; and \(3\) VLM‑based agents propose near‑uninformative experiments, failing to actively reduce hypothesis uncertainty\. To compare these results, we collect human data on the task, which reveals a gap in inductive reasoning, particularly for more complex rules\. Overall,ZendoWorldtakes an important step toward evaluating intelligent agents and identifies concrete avenues for improvement, particularly in domains like scientific discovery\. Code:[https://github\.com/ml\-research/ZendoWorld](https://github.com/ml-research/ZendoWorld) Data:[https://huggingface\.co/datasets/ss567uhg/zendo\-synthetic\-data](https://huggingface.co/datasets/ss567uhg/zendo-synthetic-data)

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.08233v1/x1.png)Figure 1:ZendoWorldcombines perception, induction, and experimentation in a controlled visual environment\.The ability to form, test, and revise hypotheses in light of new evidence is a hallmark of human intelligence, reflecting the interplay between perception and inductive reasoning\. Cognitive science has long emphasized this process as central to learning, showing how humans actively explore hypothesis spaces through informative experiments\[[10](https://arxiv.org/html/2607.08233#bib.bib93),[7](https://arxiv.org/html/2607.08233#bib.bib2),[24](https://arxiv.org/html/2607.08233#bib.bib3)\]\. From a causal perspective, this process can be understood as learning not only from observations but from interventions that probe underlying mechanisms\[[26](https://arxiv.org/html/2607.08233#bib.bib118),[27](https://arxiv.org/html/2607.08233#bib.bib119)\]\. With the growing interest in AI agents and AI scientists\[[16](https://arxiv.org/html/2607.08233#bib.bib101),[20](https://arxiv.org/html/2607.08233#bib.bib102),[38](https://arxiv.org/html/2607.08233#bib.bib100)\], structured benchmarks and controlled analyses are needed to assess these capabilities systematically\.

Recent progress in AI has focused on inductive reasoning from few\-shot examples\[[31](https://arxiv.org/html/2607.08233#bib.bib89),[19](https://arxiv.org/html/2607.08233#bib.bib99),[12](https://arxiv.org/html/2607.08233#bib.bib97),[36](https://arxiv.org/html/2607.08233#bib.bib84)\], typically evaluated on grid\-based puzzles such as ARC\-AGI\[[6](https://arxiv.org/html/2607.08233#bib.bib103)\]or visual reasoning tasks over synthetic and real images\[[3](https://arxiv.org/html/2607.08233#bib.bib107),[39](https://arxiv.org/html/2607.08233#bib.bib108),[14](https://arxiv.org/html/2607.08233#bib.bib105),[34](https://arxiv.org/html/2607.08233#bib.bib104),[30](https://arxiv.org/html/2607.08233#bib.bib109)\]\(cf\.[Figure 1](https://arxiv.org/html/2607.08233#S1.F1)blue∩\\capyellow regions\)\. While these benchmarks highlight advances in logical reasoning, particularly in the visual domain, they largely portray induction as a passive, one\-off task, thereby overlooking the iterative nature of the discovery process\.

Recent studies have begun to bridge this gap by investigating active experimentation\. For instance,\[[28](https://arxiv.org/html/2607.08233#bib.bib43)\]evaluates how LLMs can propose experiments to discover rules expressed in natural language\. Yet, in such environments, the data the systems operate on are clean symbolic abstractions, leaving out the complex nature of visual input\. Similarly, recent pushes in ARC\-AGI\-3\[[1](https://arxiv.org/html/2607.08233#bib.bib111)\]and AutumnBench\[[32](https://arxiv.org/html/2607.08233#bib.bib110)\]toward experimentation still rely on input spaces of symbolic nature \(cf\.[Figure 1](https://arxiv.org/html/2607.08233#S1.F1)blue∩\\capred\)\.

The more realistic setting lies at the intersection of active experimentation and visual grounding\. To study this, we introduceZendoWorld, an interactive benchmark inspired by the inductive logic game*Zendo*111[https://www\.looneylabs\.com/games/zendo](https://www.looneylabs.com/games/zendo)\. In this multi\-turn setting, agents observe labeled scenes, infer a hidden rule, and generate new scenes to test their hypotheses\. By combining raw visual input with a formal domain\-specific language \(DSL\) and precise feedback,ZendoWorldenables controlled evaluation of perception, induction, and experimentation within a single loop \(cf\.[Figure 1](https://arxiv.org/html/2607.08233#S1.F1)\)\.

We evaluate multiple agents on our novelZendoWorldenvironment with different degrees of Vision\-Language Model \(VLM\) integration and symbolic structure\. Our results reveal that current methods are highly specialized: while some excel at labeling or structured search, none reliably solve the full interactive loop without prior knowledge of the rule space\. Crucially, we find that labeling accuracy is often dissociated from true rule recovery; agents may correctly classify examples while remaining incapable of generating informative experiments to test their internal beliefs\.

In summary, we make the following contributions: we \(i\) introduceZendoWorld, a controlled testbed for grounded active visual rule discovery, with an instantiation spanning 22 games that jointly exercise perception, induction, and experimentation\. For this, we \(ii\) implement an evaluation protocol for hypothesis formulation, testing, and revision in visually grounded domains\. In our experiments, we \(iii\) extensively evaluate a diverse range of agents, finding that they fail to generate informative experiments and that labeling accuracy and rule recovery are dissociated\. We finally \(iv\) conduct a human study that localizes where current agents fall short of humans: humans and VLM\-based agents share surface\-level error patterns on simpler rules, but humans produce more stable rule trajectories and recover complex and out\-of\-distribution rules where no visual agent succeeds, pointing to concrete architectural directions for closing the gap\.

## 2Related Works

Table 1:Taxonomy of benchmarks at the intersection of perception, reasoning and experimentation\. Our environment \(ZendoWorld\) is the first to require grounded visual induction through active experimentation\.Benchmark / EnvPrimary ModalityPerceptionInductionExperimentationARC\-AGI\-1/2\[[6](https://arxiv.org/html/2607.08233#bib.bib103)\]Symbolic \(Grid\)×✓×Bongard, RAVEN\[[3](https://arxiv.org/html/2607.08233#bib.bib107),[39](https://arxiv.org/html/2607.08233#bib.bib108),[23](https://arxiv.org/html/2607.08233#bib.bib116)\]Graphic✓✓×Bongard\-style\[[14](https://arxiv.org/html/2607.08233#bib.bib105),[34](https://arxiv.org/html/2607.08233#bib.bib104),[21](https://arxiv.org/html/2607.08233#bib.bib114),[25](https://arxiv.org/html/2607.08233#bib.bib115)\]Visual \(3D\)✓✓×Atari, NetHack\[[17](https://arxiv.org/html/2607.08233#bib.bib113),[33](https://arxiv.org/html/2607.08233#bib.bib112)\]Visual \(2D\)✓×\*✓MineDojo\[[9](https://arxiv.org/html/2607.08233#bib.bib117)\]Visual \(3D\)✓×\*✓CLEVR\-AVR\[[40](https://arxiv.org/html/2607.08233#bib.bib94)\]Visual \(3D\)✓×\*\*✓Zendo, ActiveACRE\[[4](https://arxiv.org/html/2607.08233#bib.bib44)\]\[[28](https://arxiv.org/html/2607.08233#bib.bib43)\]Symbolic \(Text\)×✓✓ARC\-AGI\-3, AutumnBench\[[1](https://arxiv.org/html/2607.08233#bib.bib111),[32](https://arxiv.org/html/2607.08233#bib.bib110)\]Symbolic \(Grid\)×✓✓Science\-Gym\[[5](https://arxiv.org/html/2607.08233#bib.bib98)\]Symbolic & Visual \(2D\)✓✓✓ZendoWorld\(Ours\)Visual \(3D\)✓✓✓
\*RL agents typically perform implicit policy optimization rather than explicit hypothesis formulation/rule recovery\. \*\*Reasoning over images rather than explicit hypothesis formulation/rule recovery\.

Visual Induction\.A long line of work studies inductive reasoning from fixed visual examples\. Classical benchmarks such as Bongard Problems\[[3](https://arxiv.org/html/2607.08233#bib.bib107),[23](https://arxiv.org/html/2607.08233#bib.bib116)\], Raven’s Progressive Matrices\[[39](https://arxiv.org/html/2607.08233#bib.bib108)\], and ARC\-AGI\[[6](https://arxiv.org/html/2607.08233#bib.bib103)\]emphasize abstraction over synthetic or symbolic inputs, while recent Bongard variants extend to natural images and more open\-ended concepts\[[14](https://arxiv.org/html/2607.08233#bib.bib105),[34](https://arxiv.org/html/2607.08233#bib.bib104),[25](https://arxiv.org/html/2607.08233#bib.bib115),[21](https://arxiv.org/html/2607.08233#bib.bib114)\]\. As summarized in[Table 1](https://arxiv.org/html/2607.08233#S2.T1), these settings are largely passive, without allowing agents to query new examples\.

Methods in this space fall into three broad classes\. First, end\-to\-end VLMs perform few\-shot reasoning directly over images\[[15](https://arxiv.org/html/2607.08233#bib.bib79),[37](https://arxiv.org/html/2607.08233#bib.bib106)\]\. Second, neuro\-symbolic approaches combine perception with symbolic reasoning: Pix2Code\[[35](https://arxiv.org/html/2607.08233#bib.bib39)\]extracts objects and synthesizes structured concepts via DreamCoder\-style search\[[8](https://arxiv.org/html/2607.08233#bib.bib42)\], whileα\\alphaILP\[[29](https://arxiv.org/html/2607.08233#bib.bib37)\]models scenes as differentiable logic programs\. Third, Vision Language Programs \(VLP\)\[[36](https://arxiv.org/html/2607.08233#bib.bib84)\]lift visual inputs into a DSL and synthesize rules over the induced representations\.

Visual Experimentation\.A complementary line studies agents acting in visual environments, but typically under reward\-driven objectives\. Benchmarks such as Atari\[[33](https://arxiv.org/html/2607.08233#bib.bib112)\], NetHack\[[17](https://arxiv.org/html/2607.08233#bib.bib113)\], and MineDojo\[[9](https://arxiv.org/html/2607.08233#bib.bib117)\]evaluate VLM\-based policies that map observations to actions, often via prompting or hybrid RL pipelines\. Visual Agentic AI\[[22](https://arxiv.org/html/2607.08233#bib.bib38)\]extends this to programmatic spatial reasoning via APIs\. These approaches demonstrate strong perceptual control, but focus on implicit policy optimization rather than explicit rule recovery\. CLEVR\-AVR\[[40](https://arxiv.org/html/2607.08233#bib.bib94)\]moves closer to active reasoning in visual domains, but agents intervene on scenes without an explicit hypothesis space or rule representation\.

Induction through Experimentation\.A third line combines induction and experimentation, primarily in symbolic domains\. Zendo\-based setups\[[4](https://arxiv.org/html/2607.08233#bib.bib44)\]require agents to propose examples to discriminate between rules, a paradigm extended by LLM\-SMC\-S\[[28](https://arxiv.org/html/2607.08233#bib.bib43)\], which integrates LLM\-based proposal with sequential Monte Carlo inference\. Related benchmarks such as AutumnBench\[[32](https://arxiv.org/html/2607.08233#bib.bib110)\]and ARC\-AGI\-3\[[1](https://arxiv.org/html/2607.08233#bib.bib111)\]study active discovery over symbolic grids, while Science\-Gym\[[5](https://arxiv.org/html/2607.08233#bib.bib98)\]explores scientific discovery in a predefined parameter space with simple 2D scenes\.

Existing benchmarks typically cover only two of the three desired components depicted in[Table 1](https://arxiv.org/html/2607.08233#S2.T1), leaving the intersection of 3D visual grounding, rule induction, and active experimentation unstudied\.

## 3ZendoWorld

We proposeZendoWorld, an interactive reasoning benchmark inspired by the inductive logic game*Zendo*, that integrates all three components within a single evaluation loop\. The environment provides precise ground\-truth feedback and a formal DSL\-defined rule space, enabling clean separation of perception, induction, and experimentation as distinct contributors to success or failure \(cf\.[Figure 2](https://arxiv.org/html/2607.08233#S3.F2)\)\. This design supports direct comparison across method classes, including end\-to\-end VLMs, symbolic inference approaches, and program synthesis pipelines\.[Figure 2](https://arxiv.org/html/2607.08233#S3.F2)provides an overview of the game\. Briefly, an agent observes labeled visual scenes, infers a rule based on them and proposes new scenes to test and revise its hypotheses\.

![Refer to caption](https://arxiv.org/html/2607.08233v1/x2.png)Figure 2:Overview of theZendoWorldsetup\. \(A\) The agent is initialized with a set of labeled seed scenes that are consistent with a hidden rule\. \(B\) In the experimentation phase, the agent proposes new scenes, predicts their labels, and earns additional guess attempts for each correct prediction\. \(C\) When the agent decides to commit, it submits a rule hypothesis\. If incorrect, a counter example is revealed and the agent re\-enters the experimentation phase\. The episode ends when the agent guesses the rule correctly or exhausts its example budget\. Agent actions are shown in blue; environment responses in red\.### 3\.1Game Setup

AZendoWorldgame episode is defined by a latent ground\-truth ruler∗r^\{\*\}, drawn from a rule spaceℛ\\mathcal\{R\}expressed in a Prolog\-based DSL\. The episode unfolds over a number of turnst∈\{1,…\}t\\in\\\{1,\\dots\\\}, withT∈ℕT\\in\\mathbb\{N\}\. Throughout the episode, the agent accumulates a set of labeled observations𝒟=\{\(xi,yi\)\}\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}, each consisting of a visual scenexi∈ℝH×W×3x\_\{i\}\\in\\mathbb\{R\}^\{H\\times W\\times 3\}and a binary labelyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}, whereyi=1y\_\{i\}=1if the scene satisfiesr∗r^\{\*\}andyi=0y\_\{i\}=0otherwise\. The episode is bounded by the amount of total examples accumulated over the course of the game\|𝒟\|≤30\|\\mathcal\{D\}\|\\leq 30\.[Figure 3](https://arxiv.org/html/2607.08233#S3.F3)illustrates example observations for the rule*odd number of pieces*\.

![Refer to caption](https://arxiv.org/html/2607.08233v1/x3.png)Figure 3:Scene examples for the rule "odd number of pieces"\. Each scene has both a visual representation \(image\) and a symbolic representation\.Scene Representations\.Each imagexix\_\{i\}corresponds to an underlying symbolic statesi∈𝒮s\_\{i\}\\in\\mathcal\{S\}that explicitly encodes object attributes \(e\.g\., color, shape, orientation\) and their spatial arrangements \(e\.g\.,*touching*,*stacked*, or*pointing at*\)\. The latent rule is formally defined as a Boolean classifier over the symbolic space,r∗:𝒮→\{0,1\}r^\{\*\}:\\mathcal\{S\}\\rightarrow\\\{0,1\\\}, so that labels satisfyyi=r∗​\(Φ​\(xi\)\)y\_\{i\}=r^\{\*\}\(\\Phi\(x\_\{i\}\)\), whereΦ:ℝH×W×3→𝒮\\Phi:\\mathbb\{R\}^\{H\\times W\\times 3\}\\rightarrow\\mathcal\{S\}maps an image to its symbolic statesi=Φ​\(xi\)s\_\{i\}=\\Phi\(x\_\{i\}\)\. The agent must therefore bridge the high\-dimensional visual input and the logical structure of𝒮\\mathcal\{S\}to perform induction\.

Agent Interface\.At each turn, the agent observes𝒟\\mathcal\{D\}as \(image, label\) pairs\. Proposed scenesete\_\{t\}are emitted as structured specifications and rendered by the environment \([Sec\. 3\.1](https://arxiv.org/html/2607.08233#S3.SS1.SSS0.Px1)\); hypotheseshth\_\{t\}may be DSL programs or natural language\. This decouples perception from induction on the input side while keeping the action space symbolic and verifiable\.

The Interaction Loop\.The game proceeds in three phases, shown in[Figure 2](https://arxiv.org/html/2607.08233#S3.F2):

1. A\.Initialization:The agent receives two seed examples,\(xpos,1\)\(x\_\{\\text\{pos\}\},1\)and\(xneg,0\)\(x\_\{\\text\{neg\}\},0\), ensuring it starts with one positive and one negative example\.
2. B\.Experimentation:At turntt, the agent proposes an experiment by generating a new sceneet∈𝒮e\_\{t\}\\in\\mathcal\{S\}together with a predicted labely^t∈\{0,1\}\\hat\{y\}\_\{t\}\\in\\\{0,1\\\}\. The environment then returns the ground\-truth feedbackyt=r∗​\(et\)y\_\{t\}=r^\{\*\}\(e\_\{t\}\)\.
3. C\.Hypothesis Induction:The agent may propose a candidate rulehth\_\{t\}only if its prediction matched the feedback \(y^t=yt\\hat\{y\}\_\{t\}=y\_\{t\}\)\. Ifhth\_\{t\}is logically equivalent tor∗r^\{\*\}, the episode is resolved successfully\. Otherwise, a counterexamplect∈ℝH×W×3c\_\{t\}\\in\\mathbb\{R\}^\{H\\times W\\times 3\}is provided, a scene whereht​\(Φ​\(ct\)\)≠r∗​\(Φ​\(ct\)\)h\_\{t\}\(\\Phi\(c\_\{t\}\)\)\\neq r^\{\*\}\(\\Phi\(c\_\{t\}\)\), and the next turn begins again at Experimentation\.

After each turn, the new observations are appended to𝒟\\mathcal\{D\}\. The episode terminates either when\|𝒟\|\>30\|\\mathcal\{D\}\|\>30is reached or when a proposed hypothesishth\_\{t\}is logically equivalent tor∗r^\{\*\}\.

Equivalence betweenhth\_\{t\}andr∗r^\{\*\}is checked by canonicalizing both DSL programs and testing syntactic identity; natural\-language hypotheses are first translated into the DSL by an LLM\. If translation fails, an LLM judge adjudicates\. As this fallback is the least reliable step, we report its invocation rate per agent in[App\. D](https://arxiv.org/html/2607.08233#A4.SS0.SSS0.Px3)\.

##### Image Generation

Scenes are generated via a procedural Blender–Prolog pipeline\. Each scene consists of one to seven colored geometric objects \(blocks, wedges, pyramids\) with associated symbolic descriptions, yielding a combinatorial space of approximately101810^\{18\}configurations\.

All images are produced via rejection sampling\. Given a ruler∗r^\{\*\}and target labelyy, scenes are sampled uniformly and retained ifr∗​\(Φ​\(e\)\)=yr^\{\*\}\(\\Phi\(e\)\)=y\. This procedure governs both: \(i\)Seed examples, sampled conditioned on positive/negative labels, and \(ii\)Counterexamples, sampled from the discriminating set\{s∈𝒮:ht​\(s\)≠r∗​\(s\)\}\\\{s\\in\\mathcal\{S\}:h\_\{t\}\(s\)\\neq r^\{\*\}\(s\)\\\}\.

As a result, feedback consists of unbiased witnesses rather than optimized or adversarial examples, ensuring consistency across the interaction\.

##### Rule Space\.

The rule spaceℛ\\mathcal\{R\}is generated by a grammar𝒢\\mathcal\{G\}inspired by the Looney Labs Zendo card set,222[https://www\.looneylabs\.com/games/zendo](https://www.looneylabs.com/games/zendo)with predicates defined over object attributes \(color, shape, orientation\), spatial relations \(e\.g\.,*touching*,*stacked*,*pointing at*\), and quantifiers; the full specification is given in[Sec\. E\.2\.1](https://arxiv.org/html/2607.08233#A5.SS2.SSS1.Px1)\. Each rule is compiled into a program in our Prolog\-based DSL, which serves as the canonical representation used for equivalence checking and label evaluation\. The rules used during the evaluation are listed in[Table 6](https://arxiv.org/html/2607.08233#A4.T6)\.

Rules are partitioned into three categories of increasing structural complexity\.First\-Orderrules apply counting predicates such asexactlyorat\_leastto single attributes \(e\.g\.,*exactly one blue upright piece*\)\.Second\-Orderrules invoke comparative or parity quantifiers such aseven,odd,same\_amount, ormore\_than\(e\.g\.,*more red pieces than blue pieces*\)\.Compositionalrules combine these constructs with spatial relations or logical connectives \(e\.g\.,*a pyramid touching an upright piece*\)\. This taxonomy supports a stratified analysis of agent performance as a function of rule structure\.

## 4Experimental Evaluation

We structure our evaluation around four research questions:

1. Q1How well do current agents solveZendoWorldgames end\-to\-end?
2. Q2How much does perception limit performance?
3. Q3How well do agents perform rule induction, particularly as complexity increases?
4. Q4How effective is agent\-driven experimentation at reducing the hypothesis space?

### 4\.1Evaluated Agents

We evaluate four visual rule\-learning agents on challengingZendoWorldgames, spanning neural, probabilistic, and neuro\-symbolic approaches\. The agents differ primarily in their degree of structure, access to a pretrained vision model or general VLM, use of a DSL, and experimentation strategy \([Table 2](https://arxiv.org/html/2607.08233#S4.T2); details in[App\. I](https://arxiv.org/html/2607.08233#A9), prompts in[App\. J](https://arxiv.org/html/2607.08233#A10)\)\.

##### Oracle Agent \(Neuro\-Symbolic Upper Bound\)\.

TheOracle Agentserves as an approximate upper bound \([Table 2](https://arxiv.org/html/2607.08233#S4.T2), row 1\)\. It combines pretrained perception with full DSL access and performs program synthesis using DeepSynth\-style search\[[11](https://arxiv.org/html/2607.08233#bib.bib47)\], paired with heuristic experiment selection that targets maximal disambiguation between candidate rules\. We additionally consider anOracle Agent \(Random\)that replaces this strategy with random experiments, isolating the role of experimentation\.

##### VLM Agent \(End\-to\-End Neural Baseline\)\.

TheVLM Agentis the least structured approach \([Table 2](https://arxiv.org/html/2607.08233#S4.T2), row 3\), operating end\-to\-end without DSL access or explicit hypothesis search\. Rule induction and experiment selection are both handled implicitly through prompting the vision\-language model, providing a flexible but weakly constrained baseline\.

##### Bayesian Agent \(LLM\-SMC Hypothesis Inference\)\.

TheBayesian Agentintroduces structure without a DSL \([Table 2](https://arxiv.org/html/2607.08233#S4.T2), row 4\)\. It uses an LLM to propose hypotheses and maintains a particle\-based posterior via sequential Monte Carlo\[[28](https://arxiv.org/html/2607.08233#bib.bib43)\], which also guides experiment selection\. We extend prior work by incorporating image\-based prompting and allowing explicit rule proposals\.

##### Vision\-Language Programs Agent \(Program Synthesis with Learned Concepts\)\.

TheVision\-Language Programs Agentcombines VLM\-based concept extraction with program search over a general DSL \([Table 2](https://arxiv.org/html/2607.08233#S4.T2), row 5\)\. Unlike the Oracle, it does not assume predefined symbolic attributes, instead learning them from visual input\. We extend this approach to the interactive setting by enabling VLM\-guided experimentation\.

For the symbolic\-input ablation, prompts \([App\. J](https://arxiv.org/html/2607.08233#A10)\) are adapted by replacing images with Prolog\-style scene descriptions \([App\. D](https://arxiv.org/html/2607.08233#A4)\)\.

Table 2:Comparison of agent architectures\.Type describes the main architecture of the agents; DSL refers to whether agents use any DSL; Prior refers to prior knowledge over the scenes, e\.g\. whether agents use a vision model trained on the scenes; Search refers to how the hypothesis space is searched; Experimentation refers to the experimentation method used when proposing examples\.
#### 4\.1\.1Human Study

To place agent performance in the context of human capability, we collected behavioral data from 19 participants playing via a browser\-based implementation ofZendoWorlddeployed on a JATOS server\[[18](https://arxiv.org/html/2607.08233#bib.bib4)\], totaling 10 data\-points per game\. Participants completed an interactive tutorial and then played one to six games, which are a subset of the original evaluation set of games, following the same interaction protocol as the agents, with a maximum of 30 observations per episode\. Full details of the platform, interaction protocol, and data collection are provided in[App\. G](https://arxiv.org/html/2607.08233#A7)\. Results are stated in combination with the agent results\.

### 4\.2Experimental Setup

Data\.We consider a set of2222games ofZendoWorldwith different hidden rules spanning basic predicates, counting, parity, spatial relations, and logical connectives plus one out\-of\-distribution \(OOD\) rule that cannot be expressed in the DSL \(cf\.[Table 6](https://arxiv.org/html/2607.08233#A4.T6)\)\. All agents use a common episode protocol with a maximum of\|𝒟\|=30\|\\mathcal\{D\}\|=30observations per game, and the same interaction loop\.

Models\.For theOracle Agentwe use a pretrained vision model specified in[App\. E](https://arxiv.org/html/2607.08233#A5)\. For the other agents that rely on visual input, we fix the VLM backbone togpt\-5\-mini\-2025\-08\-07to ensure consistent cross\-method comparison\. We report averages over multiple runs where applicable\. Ablations using different VLM\-backbones are given in[Table 5](https://arxiv.org/html/2607.08233#A2.T5)\.

Metrics\.To answer Q1 we report the absolute number of games won and the win rate for the agents\. A game is won if the final hypothesis is logically equivalent to the hidden rule\. We also report the average turn count, computed over solved episodes only\. For Q3 we additionally investigate intermediate hypotheses produced by each agent\. We represent each hypothesis as a symbolic program and compare it to the ground\-truth rule using a structural F1 score over the rule tree\. This metric measures overlap between predicates and logical operators under a permutation\-invariant matching of commutative subexpressions\. Finally, for Q4 we use anexpected information gain\(EIG\) metric that measures, relative to the current hypothesis posterior, how much uncertainty over possible rules a given example would resolve\. The full EIG definition and experimental details are given in[App\. C](https://arxiv.org/html/2607.08233#A3)\.

Environmental Footprint\.All experiments ran on a single NVIDIA Tesla V100\-SXM3\-32GB GPU\. The complete evaluation \(22 games×\\times5 seeds×\\times4 agents = 440 runs\) totaled 321 GPU\-hours \(13\.375 days wall\-clock\) and approximately 155M GPT\-5\-mini API tokens\. Per\-agent runtime and token details are provided in[App\. A](https://arxiv.org/html/2607.08233#A1)\.

### 4\.3Experimental Evaluations

Table 3:Agent evaluation results on ZendoWorld\.Results are averaged across seeds and games±\\pmSEM across seeds\.Bestandsecond bestvalues per column highlighted\.Shaded row indicates Oracle Agent with privileged access to the DSL rule space\.![Refer to caption](https://arxiv.org/html/2607.08233v1/x4.png)
Figure 4:Win rate per complexity class\.Humans excel over AI Agents, the gap becomes especially large for more challenging tasks\. Lines show SEM across seeds and tasks\.
ZendoWorldgames are challenging for agents \(Q1\)\.We evaluate current AI agents on the 22ZendoWorldgames to assess their ability to solve the full perception–induction loop in[Sec\. 4\.3](https://arxiv.org/html/2607.08233#S4.SS3), comparing them with the human performance on the subset of games\. We see that theOracle Agentachieves strong results, solving 95\.5% of games with an average of 5\.2 turns\. While this performance is high, it is expected given access to pretrained perception modules and the ground\-truth DSL\. Its single failure occurs on the OOD task, which lies outside its hypothesis space, highlighting a key limitation: the Oracle is effective only when the true rule is expressible in its DSL\.

Human participants perform better than the VLM\-based agents in terms of win rate and require a similar amount of turns but achieve lower label accuracy \(53\.9%\)\. Notably, 7 out of 10 participants that played the OOD task solved it, whereas no agent does on visual inputs \([App\. H](https://arxiv.org/html/2607.08233#A8)\)\. This task involves second\-order logic, which poses a particular challenge for VLM\-based agents and exposes weaknesses in their rule induction\.

Among agents without prior environment\-specific knowledge, the pure VLM\-based agent performs best, solving 44\.5% of games on average\. In contrast, theVLP and Bayesian Agentsolve 18\.2% and 13\.6%, respectively, despite being designed to improve over direct prompting\. All learned agents require substantially more turns than the Oracle and still fail on many episodes, indicating inefficient exploration and hypothesis refinement\. To better understand the role of the VLM backbone, we evaluate the VLM\-based agents with several state\-of\-the\-art models in[App\. B](https://arxiv.org/html/2607.08233#A2)\. While we discover interesting differences, across all settings, agent performance remains far below human\-level play\.

Overall,current methods do not robustly solve the full task, despite strong performance on individual components\. TheBayesian Agent, for instance, achieves the highest label accuracy during interaction \(75\.2%\), consistent with prior work\[[28](https://arxiv.org/html/2607.08233#bib.bib43)\], yet this does not translate into end\-to\-end success\. This supports recent findings that correct classification alone does not entail recovering of the underlying rules\[[2](https://arxiv.org/html/2607.08233#bib.bib86)\]\.

The effect of rule complexity further highlights these limitations, shown in[Figure 4](https://arxiv.org/html/2607.08233#S4.F4)\. TheVLM Agentsolves 67\.5% of first\-order logic games but drops to approximately 30% on second\-order and complex games\. TheBayesian Agentsucceeds only on first\-order logic games and fails entirely on more complex settings\. In contrast, theVLP Agentmaintains relatively consistent performance across first\-order and second\-order games\. Human participants achieve at least a 50% win rate across all game types\. This indicates agap between VLM\-based rule induction and human\-like rule learning,particularly as rule complexity increases\. TheVLM Agentsolves 40% of the six tasks solved by the humans whereas the participants achieve 73\.3% highlighting the gap in rule induction\.

Table 4:Agent evaluation results on SymbolicZendoWorld\.Results are averaged across seeds and games±\\pmSEM across seeds\. Green/red annotations show relative change vs\. the perception\-based variant\.Bestandsecond bestper column highlighted\.Shaded row indicates oracle agent with privileged access to the DSL rule space\.Perception is a bottleneck for inducing correct rules \(Q2\)\.To disentangle perception from induction, we evaluate all agents in a symbolic\-input setting, replacing rendered scenes with ground\-truth Prolog\-style descriptions \([Table 4](https://arxiv.org/html/2607.08233#S4.T4)\)\.

Removing visual input improves performance for all learned agents, confirming that perception is a key bottleneck\. TheVLP Agentbenefits most, improving by over160%160\\%and solving nearly half of the games\. TheVLM Agentshows a more moderate gain of29\.4%29\.4\\%, likely due to its relatively strong baseline performance\. TheBayesian Agentalso improves, but remains the weakest overall, solving fewer than one third of the games\. Part of the performance gap may stem from the mismatch between Blender\-generated scenes and the natural images used during pretraining, limiting generalization in visual reasoning\.

Despite these gains, no agent approaches Oracle\-level performance, indicating that induction remains a fundamental bottleneck even when perception is removed\. A breakdown by rule category \([Table 6](https://arxiv.org/html/2607.08233#A4.T6)\) shows that VLM\-based agents struggle particularly with second\-order and compositional rules\. TheVLP Agentis the only method that solves the OOD task, likely due to its more expressive DSL \(e\.g\., comparative operators\), although this flexibility incurs a higher search cost\. Collectively, these results suggest thatboth perception and induction constrain performance, with their relative impact varying across methods\.

![Refer to caption](https://arxiv.org/html/2607.08233v1/x5.png)Figure 5:Smoothed structural F1 score by number of observed examples at each rule guess, averaged across 22 episodes \(5 seeds\)\. Left: human gameruns\. Right: agent gameruns stratified by rule type\. Shaded bands show±1\\pm 1SEM across seeds\. The guesses come from the primary experiment using visual zendo scenes\. The top plots show only the guesses from games where agents won, bottom plots show experiments from all games\. The x‑axis indicates the number of examples seen; agents begin guessing rules from the fourth example onward\. To remove length effects, the final F1 of solved games is carried forward\.Increased Rule Complexity Reduces Convergence Speed \(Q3\)\.To better understand the space of hypothesis the agents explore, we compute a structural F1 curve per game as a function of the number of observed examples, and then average these curves across games to obtain the aggregate learning trajectory\.[Figure 5](https://arxiv.org/html/2607.08233#S4.F5)shows the structural F1 score of intermediate rule guesses as a function of the number of observed examples\.

In the top plot, we restrict to successful episodes, isolating convergence independent of final game outcomes\. On first\-order logic games, human participants converge at a rate similar to theVLM Agent, while on higher\-complexity tasks their performance more closely matches theOracle Agent\. Notably, humans converge more slowly on second\-order logic tasks than on complex tasks, whereas VLM\-based agents show the opposite trend, likely reflecting difficulties in perceiving relations between objects\.

The bottom plots reveal a late\-stage drop in F1 for both humans and VLM\-based agents\. This reflects a common failure mode: after proposing a near\-correct rule and receiving a counterexample, agents overcorrect and move further from the true rule\. For instance, in a game with the rule“at least three red objects and exactly one pyramid”, both a human participant and theVLM Agentproduce an almost\-correct hypothesis by the 7th guess, but subsequent revisions diverge further from the target\. Overall,higher rule complexity slows convergenceand induces shared overcorrection behaviors in both humans and learned agents\.

Strong inductive abilities improve informativeness of experiments \(Q4\)\.To assess how proposed experiments drive convergence, we analyze agents in the symbolic setting and introduce anOracle Agent \(Random\)variant that proposes random scenes\.

We measure the expected information gain \(EIG\) of proposed experiments\. As shown in[Figure 6](https://arxiv.org/html/2607.08233#S4.F6), VLM\-based agents achieve substantially lower EIG, suggesting they generate examples similar to prior observations and contribute little new information\. In contrast, theOracle Agentmaximizes EIG by selecting experiments that best discriminate between competing hypotheses\. Notably, both theVLM\- and Bayesian Agentsexhibit near\-zero EIG after roughly 10 examples, indicating reliance on already refuted hypotheses or insufficiently diverse proposals\.

![Refer to caption](https://arxiv.org/html/2607.08233v1/x6.png)Figure 6:Mean expected information gain \(EIG, bits\) by example position of experiment within the game, averaged across tasks and seeds\. Shaded bands show±1\\pm 1SE\. The data is from the secondary experiment using symbolic input instead of images\.Interestingly, theOracle Agent \(Random\)achieves the second\-highest EIG, suggesting that diverse scene proposals can reduce the search space more effectively than hypothesis\-guided search based on incorrect hypotheses\. The low ranking of theBayesian Agent, despite selecting the best of 10 candidates, further supports this interpretation:a poorly calibrated hypothesis set can actively misguide experiment selection\.

However, a diverse set of hypotheses makes it easier to find good experiments that reduce uncertainty\. This suggests that agents should adapt their strategy to rely more on diverse or random exploration when uncertain and shift toward hypothesis\-guided experimentation only once their beliefs become reliable\.

Discussion & Future Work\.The results reveal distinct failure modes across agents\. TheOracle Agentsucceeds via its tailored perception module and DSL access, but fails when the hidden rule requires an out\-of\-vocabulary predicate\. Among VLM\-based methods, theVLM Agentperforms best end\-to\-end yet still fails on over half of episodes, especially on relational and second\-order rules\. TheBayesian Agentexcels at labeling but only recovers short, simple rules; merging hypotheses via logical connectives and providing more observations in the prompt could push it toward more complex ones\. TheVLP Agenthandles predicates like EVEN/ODD that others miss, but structured search alone is insufficient when the predicate inventory is incomplete or the generalized DSL inflates the search space\. Overall,ZendoWorldexposes a clear gap between solving individual subcomponents and the full interactive loop\.

A deeper architectural question concerns the strict separation between perception and reasoning itself\. TheOracle Agent’s modular design is interpretable and efficient, but its perceptual vocabulary is fixed before rule learning begins: the DSL predicates define not only what can be reasoned about, but also what can be*seen*\. Hofstadter\[[13](https://arxiv.org/html/2607.08233#bib.bib92)\]argues that human conceptual thinking does not proceed with such clean separation; perception and abstract reasoning are mutually constitutive, with active concepts reshaping attended features and novel groupings giving rise to new distinctions\. A more principled Zendo player might therefore embed perception within the hypothesis\-testing loop, letting the current best hypothesis direct visual attention toward features most relevant for discriminating among remaining candidates\.

## 5Conclusion

We introduced the benchmarkZendoWorld, a challenging, controlled interactive environment that integrates perception, induction, and experiment design within a single closed loop, and used it to investigate the core challenges of active visual rule learning\. Our results establish three main findings\. First, strong performance on example labeling does not guarantee correct hypothesis induction, with important implications for how inductive reasoning systems should be evaluated\. Second, perception and induction failures are distinct and agent\-dependent: removing visual input substantially improves theVLP Agent, but has little effect on theBayesian\- and VLM Agent, suggesting that their primary bottleneck lies in induction and hypothesis revision rather than perception\. Third, VLM\-based agents consistently fail to actively reduce hypothesis uncertainty, proposing experiments with near\-zero expected information gain even when perception is removed\. This persistent failure points to a structural limitation in how these agents explore and search the hypothesis space\.

These findings suggest that progress in active rule learning will require architectures that combine visual grounding with structured, falsifiable hypotheses and active search\. Two promising directions are: \(1\) tighter feedback loops between hypotheses and observations, allowing agents to revisit examples and resolve inconsistencies, and \(2\) hybrid symbolic\-neural systems where structured hypotheses guide perceptual feature extraction instead of relying on a fixed perception module\.

## Acknowledgments

This project was partially funded by the Deutsche Forschungsgemeinschaft \(DFG, German Research Foundation\) under Germany´s Excellence Strategy \(EXC\-3057/1 „Reasonable Artificial Intelligence“, Project No\. 533677015\)\. This work was further supported by the Priority Program \(SPP\) 2422 in the subproject “Optimization of active surface design of high\-speed progressive tools using machine and deep learning algorithms“ funded by the German Research Foundation \(DFG\)\. It has also benefited from the Cluster of Excellence "The Adaptive Mind", funded by the DFG under Germany’s Excellence Strategy \- EXC\-3066\.

## References

- \[1\]ARC Prize Foundation\(2026\)ARC\-AGI\-3: a new challenge for frontier agentic intelligence\.arXiv preprint arXiv:2603\.24621\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p3.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.9.9.1),[§2](https://arxiv.org/html/2607.08233#S2.p4.1)\.
- \[2\]C\. Beger, R\. Yi, S\. Fu, K\. Denton, A\. Moskvichev, S\. W\. Tsai, S\. Rajamanickam, and M\. Mitchell\(2025\)Do ai models perform human\-like abstract reasoning across modalities?\.arXiv preprint arXiv:2510\.02125\.Cited by:[§4\.3](https://arxiv.org/html/2607.08233#S4.SS3.p4.1)\.
- \[3\]M\. M\. BongardJ\.K\. Hawkins \(Ed\.\)\(1970\)Pattern Recognition\.Spartan Books\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.3.3.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[4\]N\. R\. Bramley, A\. Rothe, J\. B\. Tenenbaum, F\. Xu, and T\. M\. Gureckis\(2018\)Grounding compositional hypothesis generation in specific instances\.InProceedings of the Annual Meeting of the Cognitive Science Society,Vol\.40\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.8.8.1),[§2](https://arxiv.org/html/2607.08233#S2.p4.1)\.
- \[5\]M\. Cerrato, L\. Baur, J\. Brugger, S\. Shumaly, N\. Schmitt, E\. Finkelstein, S\. Jukic, L\. Münzel, F\. P\. Paul, P\. Pfannes,et al\.\(2026\)Science\-gym: a simple testbed for ai\-driven scientific discovery\.Machine Learning115\(1\),pp\. 16\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.10.10.1),[§2](https://arxiv.org/html/2607.08233#S2.p4.1)\.
- \[6\]F\. Chollet\(2019\)On the measure of intelligence\.arXiv preprint arXiv:1911\.01547\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.2.2.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[7\]C\. Cook, N\. D\. Goodman, and L\. E\. Schulz\(2011\)Where science starts: spontaneous experiments in preschoolers’ exploratory play\.Cognition120\(3\),pp\. 341–349\.Note:Probabilistic models of cognitive developmentExternal Links:ISSN 0010\-0277Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[8\]K\. Ellis, L\. Wong, M\. Nye, M\. Sable\-Meyer, L\. Cary, L\. Anaya Pozo, L\. Hewitt, A\. Solar\-Lezama, and J\. B\. Tenenbaum\(2023\)DreamCoder: growing generalizable, interpretable knowledge with wake–sleep bayesian program learning\.Philosophical Transactions of the Royal Society A381\(2251\),pp\. 20220050\.Cited by:[§2](https://arxiv.org/html/2607.08233#S2.p2.1)\.
- \[9\]L\. Fan, G\. Wang, Y\. Jiang, A\. Mandlekar, Y\. Yang, H\. Zhu, A\. Tang, D\. Huang, Y\. Zhu, and A\. Anandkumar\(2022\)Minedojo: building open\-ended embodied agents with internet\-scale knowledge\.Advances in Neural Information Processing Systems35,pp\. 18343–18362\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.6.6.1),[§2](https://arxiv.org/html/2607.08233#S2.p3.1)\.
- \[10\]G\. J\. Feist and M\. E\. Gorman\(1998\)The psychology of science: review and integration of a nascent discipline\.Review of General Psychology2,pp\. 3 – 47\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[11\]N\. Fijalkow, G\. Lagarde, T\. Matricon, K\. Ellis, P\. Ohlmann, and A\. N\. Potta\(2022\)Scaling neural program synthesis with distribution\-based search\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 6623–6630\.Cited by:[Appendix C](https://arxiv.org/html/2607.08233#A3.p2.8),[§E\.2](https://arxiv.org/html/2607.08233#A5.SS2.p1.1),[Appendix E](https://arxiv.org/html/2607.08233#A5.p1.1),[§4\.1](https://arxiv.org/html/2607.08233#S4.SS1.SSS0.Px1.p1.1)\.
- \[12\]L\. Helff, A\. Omar, F\. Friedrich, A\. Wüst, H\. Shindo, R\. Mitchell, T\. Woydt, P\. Schramowski, W\. Stammer, and K\. Kersting\(2026\)SLR: automated synthesis for scalable logical reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(ACL 2026\),Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1)\.
- \[13\]D\. R\. Hofstadter\(1995\)Fluid concepts and creative analogies: computer models of the fundamental mechanisms of thought\.\.Basic books\.Cited by:[§4\.3](https://arxiv.org/html/2607.08233#S4.SS3.p17.1)\.
- \[14\]H\. Jiang, X\. Ma, W\. Nie, Z\. Yu, Y\. Zhu, and A\. Anandkumar\(2022\)Bongard\-HOI: benchmarking few\-shot visual reasoning for human\-object interactions\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.4.4.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[15\]O\. V\. Johnson, O\. M\. Alyasiri, D\. Akhtom, and O\. E\. Johnson\(2023\)Image analysis through the lens of chatgpt\-4\.Journal of Applied Artificial Intelligence4\(2\),pp\. 31–46\.Cited by:[§2](https://arxiv.org/html/2607.08233#S2.p2.1)\.
- \[16\]R\. D\. King, J\. Rowland, W\. Aubrey, M\. Liakata, M\. Markham, L\. N\. Soldatova, K\. E\. Whelan, A\. Clare, M\. Young, A\. Sparkes,et al\.\(2009\)The robot scientist adam\.Computer42\(8\),pp\. 46–54\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[17\]H\. Küttler, N\. Nardelli, A\. Miller, R\. Raileanu, M\. Selvatici, E\. Grefenstette, and T\. Rocktäschel\(2020\)The nethack learning environment\.Advances in Neural Information Processing Systems33,pp\. 7671–7684\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.5.5.1),[§2](https://arxiv.org/html/2607.08233#S2.p3.1)\.
- \[18\]K\. Lange, S\. Kühn, and E\. Filevich\(2015\)" Just another tool for online studies”\(jatos\): an easy solution for setup and management of web servers supporting online studies\.PloS one10\(6\),pp\. e0130834\.Cited by:[§4\.1\.1](https://arxiv.org/html/2607.08233#S4.SS1.SSS1.p1.1)\.
- \[19\]W\. Li, K\. Hu, C\. Larsen, Y\. Wu, S\. Alford, C\. Woo, S\. M\. Dunn, H\. Tang, W\. Zheng, Y\. Pu, and K\. Ellis\(2025\)Combining induction and transduction for abstract reasoning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1)\.
- \[20\]C\. Lu, C\. Lu, R\. Lange, J\. Foerster, J\. Clune, and D\. Ha\(2024\)The ai scientist: towards fully automated open\-ended scientific discovery\.ArXivabs/2408\.06292\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[21\]M\. Małkiński, S\. Pawlonka, and J\. Mańdziuk\(2025\)Reasoning limitations of multimodal large language models\. a case study of bongard problems\.InForty\-second International Conference on Machine Learning,Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.4.4.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[22\]D\. Marsili, R\. Agrawal, Y\. Yue, and G\. Gkioxari\(2025\)Visual agentic ai for spatial reasoning with a dynamic api\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 19446–19455\.Cited by:[§2](https://arxiv.org/html/2607.08233#S2.p3.1)\.
- \[23\]W\. Nie, Z\. Yu, L\. Mao, A\. B\. Patel, Y\. Zhu, and A\. Anandkumar\(2020\)Bongard\-logo: a new benchmark for human\-level concept learning and reasoning\.Advances in neural information processing systems33,pp\. 16468–16480\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.3.3.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[24\]M\. Oaksford and N\. Chater\(1994\)A rational analysis of the selection task as optimal data selection\.\.Psychological review101\(4\),pp\. 608\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[25\]S\. Pawlonka, M\. Małkiński, and J\. Mańdziuk\(2025\)Bongard\-rwr\+: real\-world representations of fine\-grained concepts in bongard problems\.arXiv preprint arXiv:2508\.12026\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.4.4.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[26\]J\. Pearl\(2009\)Causality\.Cambridge university press\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[27\]J\. Pearl\(2018\)The book of why: the new science of cause and effect\.Basic Books\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[28\]T\. Piriyakulkij, C\. Langenfeld, T\. A\. Le, and K\. Ellis\(2024\)Doing experiments and revising rules with natural language and probabilistic reasoning\.Advances in Neural Information Processing Systems37,pp\. 53102–53137\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p3.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.8.8.1),[§2](https://arxiv.org/html/2607.08233#S2.p4.1),[§4\.1](https://arxiv.org/html/2607.08233#S4.SS1.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2607.08233#S4.SS3.p4.1)\.
- \[29\]H\. Shindo, V\. Pfanschilling, D\. S\. Dhami, and K\. Kersting\(2023\)α\\alphaIlp: thinking visual scenes as differentiable logic programs\.Machine Learning112\(5\),pp\. 1465–1497\.Cited by:[§2](https://arxiv.org/html/2607.08233#S2.p2.1)\.
- \[30\]D\. Steinmann, W\. Stammer, A\. Wüst, and K\. Kersting\(2025\)Object centric concept bottlenecks\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1)\.
- \[31\]R\. Wang, E\. Zelikman, G\. Poesia, Y\. Pu, N\. Haber, and N\. Goodman\(2023\)Hypothesis search: inductive reasoning with language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1)\.
- \[32\]A\. Warrier, D\. Nguyen, M\. Naim, M\. Jain, Y\. Liang, K\. Schroeder, C\. Yang, J\. B\. Tenenbaum, S\. Vollmer, K\. Ellis,et al\.\(2025\)Benchmarking world\-model learning\.arXiv preprint arXiv:2510\.19788\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p3.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.9.9.1),[§2](https://arxiv.org/html/2607.08233#S2.p4.1)\.
- \[33\]N\. R\. Waytowich, D\. White, M\. Sunbeam, and V\. G\. Goecks\(2024\)Atari\-gpt: benchmarking multimodal large language models as low\-level policies in atari games\.arXiv preprint arXiv:2408\.15950\.Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.5.5.1),[§2](https://arxiv.org/html/2607.08233#S2.p3.1)\.
- \[34\]R\. Wu, X\. Ma, Z\. Zhang, W\. Wang, Q\. Li, S\. Zhu, and Y\. Wang\(2024\)Bongard\-OpenWorld: few\-shot reasoning for free\-form visual concepts in the real world\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.4.4.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[35\]A\. Wüst, W\. Stammer, Q\. Delfosse, D\. S\. Dhami, and K\. Kersting\(2024\)Pix2Code: learning to compose neural visual concepts as programs\.InUncertainty in Artificial Intelligence,pp\. 3829–3852\.Cited by:[§2](https://arxiv.org/html/2607.08233#S2.p2.1)\.
- \[36\]A\. Wüst, W\. Stammer, H\. Shindo, D\. S\. Dhami, L\. Helff, and K\. Kersting\(2026\)Synthesizing visual concepts as vision\-language programs\.IEEE/CVF Conference on Computer Vision and Pattern Recognition\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1),[§2](https://arxiv.org/html/2607.08233#S2.p2.1)\.
- \[37\]A\. Wüst, T\. Tobiasch, L\. Helff, I\. Ibs, W\. Stammer, D\. S\. Dhami, C\. A\. Rothkopf, and K\. Kersting\(2025\)Bongard in wonderland: visual puzzles that still make AI go mad?\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2607.08233#S2.p2.1)\.
- \[38\]Q\. Xie, Y\. Weng, M\. Zhu, F\. Shen, S\. Huang, Z\. Lin, J\. Zhou, Z\. Mao, Z\. Yang, L\. Yang,et al\.\(2025\)How far are ai scientists from changing the world?\.arXiv preprint arXiv:2507\.23276\.Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p1.1)\.
- \[39\]C\. Zhang, F\. Gao, B\. Jia, Y\. Zhu, and S\. Zhu\(2019\)RAVEN: A dataset for relational and analogical visual reasoning\.InConference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2607.08233#S1.p2.1),[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.3.3.1),[§2](https://arxiv.org/html/2607.08233#S2.p1.1)\.
- \[40\]W\. Zhou, X\. Xiong, Y\. Peng, M\. Tao, C\. Zhao, H\. Dong, M\. Tang, and J\. Wang\(2025\)PhysVLM\-avr: active visual reasoning for multimodal large language models in physical environments\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[Table 1](https://arxiv.org/html/2607.08233#S2.T1.5.7.7.1),[§2](https://arxiv.org/html/2607.08233#S2.p3.1)\.

## Appendix AEnvironmental Footprint

For the primary experiment, we collected data from 22 games × 5 seeds × 4 agents = 440 runs;

- •Oracle Agent: 30h 50m \(2m 55s/turn, 632 turns\)
- •VLM Agent: 59h 28m \(2m 21s/turn, 1509 turns\), 22,875,048 Tokens
- •Bayesian: 102h 47m \(3m 43s/turn, 1655 turns\), 115,242,419 Tokens
- •VLP Agent: 128h 46m \(4m 32s/turn, 1699 turns\), 17,007,727 Tokens

Total:321 GPU\-hours \(13\.375 days wall\-clock\), 155,125,194 GPT\-5\-mini API token usage\.

## Appendix BBackbone Ablations

To assess the impact of the underlying Vision\-Language Model \(VLM\) backbone on agent performance, we conduct an ablation study replacing GPT\-5\-mini with several recent open\-source and proprietary models\. Specifically, we evaluate both the VLM and VLP agents on the same set of2222games using Qwen3\.5\-27B, Gemma\-4\-31B\-it, claude\-opus\-4\-8 and gemini\-3\.1\-pro\-preview, denoted as \(Qwen\), \(Gemma\), \(Claude\) and \(Gemini\), respectively, in[Table 5](https://arxiv.org/html/2607.08233#A2.T5)\.

Table 5:Comparison of VLM and VLP agents across five backbone models \(GPT, Gemma, Qwen, Claude, Gemini\)\.Wins \(%\)reports the agent’s win rate across evaluation episodes,Avg Turnsthe average turns taken to win the game, andLabel Acc \(%\)the accuracy of predicted labels against ground truth\.Boldandunderlinedvalues indicate the best and second\-best result per column, respectively\. Values are reported as mean±\\pmSEM\.We observe that the fully end\-to\-end VLM Agent exhibits substantial performance degradation when using smaller open\-source backbones\. These models frequently fail to produce syntactically valid or parseable rules and show increased error rates in example labeling\. Such failures compound over the interaction trajectory, substantially reducing the likelihood of correctly inferring the target rule\. Among the large proprietary models, Claude shows a surprisingly low win rate\. Gemini achieves a slightly higher win rate than GPT while requiring substantially fewer turns\. However, none of these state\-of\-the\-art models come close to solving all ZendoWorld puzzles, and their performance remains far from human\-level play\.

In contrast, the VLP Agent shows a different pattern\. Interestingly, VLM size does not appear to correlate strongly with performance: the VLP Agent remains competitive across backbones and, in several metrics, improves when paired with alternative models\. This robustness likely stems from its modular design\. The VLM is used primarily for scene perception, while structured reasoning is handled downstream, reducing reliance on the model’s ability to generate complex, well\-formed symbolic rules\.

## Appendix CExpected Information Gain

To quantify the informativeness of player\-proposed examples, we compute the*expected information gain*\(EIG\) of each example relative to the current hypothesis posterior\.

Givennnprior labeled examplesE=\{\(xi,yi\)\}i=1nE=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, we enumerate the top\-KKK=20K=20most probable programsH=\{hj\}j=1K⊆ℋH=\\\{h\_\{j\}\\\}\_\{j=1\}^\{K\}\\subseteq\\mathcal\{H\}consistent withEEusing heap search\[[11](https://arxiv.org/html/2607.08233#bib.bib47)\]over a uniform Probabilistic Context Free Grammar \(PCFG\) derived from the DSL grammar\. Tasks for which the search finds fewer than 2 perfectly consistent programs are excluded from the EIG analysis, as the hypothesis distribution is fully determined, this typically occurs when the examples fully specify a single rule, such that any alternative rule is inconsistent with the observed data\. This happens for short rules towards the end of the game when more than 20 examples have been added\. Eachhjh\_\{j\}has a prior weight given by the PCFGwj=P​\(hj\)w\_\{j\}=P\(h\_\{j\}\)\. We form a posterior over the retained hypotheses by renormalizing these weights:

P​\(hj∣E\)=wj⋅1​\[hj​\(⋅\)⊧E\]∑hkwk⋅1​\[hk​\(⋅\)⊧E\],P\(h\_\{j\}\\mid E\)=\\frac\{w\_\{j\}\\cdot 1\[h\_\{j\}\(\\cdot\)\\models E\]\}\{\\sum\_\{h\_\{k\}\}w\_\{k\}\\cdot 1\[h\_\{k\}\(\\cdot\)\\models E\]\},ensuring that the probability mass sums to 1 over the top\-KKconsistent hypotheses\. For a proposed examplex∗x^\{\*\}, the EIG is the binary entropy of the predicted label:

EIG​\(x∗\)=H​\(∑j=1KP​\(hj∣E\)⋅1​\[hj​\(x∗\)=True\]\),\\text\{EIG\}\(x^\{\*\}\)=H\\\!\\left\(\\sum\_\{j=1\}^\{K\}P\(h\_\{j\}\\mid E\)\\cdot 1\[h\_\{j\}\(x^\{\*\}\)=\\text\{True\}\]\\right\),whereH​\(p\)=−p​log2⁡p−\(1−p\)​log2⁡\(1−p\)H\(p\)=\-p\\log\_\{2\}p\-\(1\-p\)\\log\_\{2\}\(1\-p\)\. EIG≈1\\approx 1when hypotheses split 50/50 \(maximally informative\); EIG≈0\\approx 0when all agree \(redundant\)\. We report average EIG per agent as a measure of sample efficiency\.

## Appendix DDataset Details

We generated a synthetic dataset of labeled Zendo scenes using Prolog and Blender shown in[7\(a\)](https://arxiv.org/html/2607.08233#A4.F7.sf1)\. Each scene depicts a structure composed of colored geometric shapes \(blocks, wedges, pyramids\) in different orientations, with up to seven pieces per scene\. The data generation pipeline samples random logical rules from predefined templates involving logical combinators, quantifiers, attributes, and relations\. Each rule is automatically translated into a Prolog query to generate examples that either satisfy or violate the rule\.[7\(b\)](https://arxiv.org/html/2607.08233#A4.F7.sf2)shows an example overlayed with the bounding box output of the model and linked with the color head output\.

![Refer to caption](https://arxiv.org/html/2607.08233v1/x7.png)\(a\)Generation pipeline\.
![Refer to caption](https://arxiv.org/html/2607.08233v1/figures/color_classification.png)\(b\)Model outputs for color head\.

Figure 7:Overview of the generation pipeline and example model output\.##### Object Detection Data:

We generated 54,252 training images at640×480640\\times 480resolution, each annotated with per\-object attributes \(color, shape, orientation, bounding box\) and relations \(pointing,touching,on\_top\_of\)\. Object encodings follow the vector format:

\[𝙸𝙳,𝙲𝙾𝙻𝙾𝚁,𝚂𝙷𝙰𝙿𝙴,𝙾𝚁𝙸𝙴𝙽𝚃𝙰𝚃𝙸𝙾𝙽,𝚕𝚎𝚏𝚝,𝚛𝚒𝚐𝚑𝚝,𝚏𝚛𝚘𝚗𝚝,𝚋𝚊𝚌𝚔,𝚝𝚘𝚙,𝚋𝚘𝚝𝚝𝚘𝚖,𝚙𝚘𝚒𝚗𝚝𝚒𝚗𝚐,xmin,ymin,xmax,ymax\]\[\\mathtt\{ID\},\\mathtt\{COLOR\},\\mathtt\{SHAPE\},\\mathtt\{ORIENTATION\},\\mathtt\{left\},\\mathtt\{right\},\\\\ \\mathtt\{front\},\\mathtt\{back\},\\mathtt\{top\},\\mathtt\{bottom\},\\mathtt\{pointing\},x\_\{\\min\},y\_\{\\min\},x\_\{\\max\},y\_\{\\max\}\]\(1\)
A separate test set of 2,779 images was generated with comparable attribute distributions \(e\.g\.,52%52\\%vs\.54%54\\%touchingrelations\)\. Each categorical attribute is mapped to numerical encodings, for example:color: red = 0, blue = 1, yellow = 2;shape: block = 0, wedge = 1, pyramid = 2\.

##### Program Synthesis Data:

For rule induction, we created 1,700 games from 928 unique rules drawn from the official Zendo card set by Looney Labs333[https://www\.looneylabs\.com/games/zendo](https://www.looneylabs.com/games/zendo)\. Each game pairs a DSL rule with 20 labeled examples \(10 positive, 10 negative\), generated via the Prolog pipeline\. The Rules take object vectors from \([1](https://arxiv.org/html/2607.08233#A4.E1)\) as input \(omitting bounding boxes, unused for evaluation\)

We categorize games by rule complexity:

First Order:Rules including predicates:EXACTLY,AT\_LEAST,EXCLUSIVELY,ZERO

Second Order:Rules including predicates:EVEN,ODD,MORE\_THAN

Complex:Rules describing relations \(POINTING,TOUCHING,ON\_TOP\_OF\) or combining multiple sub\-rules using \(AND,OR\)\.

Additionally, we include a rule that includes the predicateSAME\_AMOUNTthat acts as an out\-of\-distribution task for the Oracle Agent where we deliberately left out the predicate in its DSL\.

We display all episodes in[Table 6](https://arxiv.org/html/2607.08233#A4.T6)with the complexity level and the win rate per agent\.

Table 6:All 22 Zendo games sorted by complexity, with the win rate per Player/Agent\.
##### Rule Guess Parsing\.

When an agent’s hypothesis cannot be parsed into a valid DSL program, the system falls back to querying an LLM for logical equivalence and counterexample generation instead of using the algorithmic comparison\. This fallback is rare across all agents: the VLP Agent triggered it in 68 of 1,083 guesses \(6\.3%\), the VLM Agent in 34 of 665 \(5\.1%\) and the Bayesian Agent in 18 of 1,020 \(1\.8%\)\. The low fallback rates indicate that the evaluation pipeline relies on formal program comparison in the vast majority of cases\.

## Appendix EModel

The base of the Neuro\-Symbolic Zendo player consists of two separate modules that are connected through a shared encoding of outputs and inputs\. The first is a vision model that detects each piece in a Zendo structure\. This model is trained on synthetic images generated specifically for the task\. Its output is a list of vectors, each representing one piece in the structure\. The second module is a program synthesizer based on DeepSynth\[[11](https://arxiv.org/html/2607.08233#bib.bib47)\]\. Its inputs are input\-output\-pairs \(I/O examples\) in the form of lists of vectors and labels\. The labels indicate whether a specific structure follows the hidden rule\. Given several I/O examples, the synthesizer produces a program that classifies structures accordingly\. This program can be interpreted directly as a logical rule\.

### E\.1Vision Model

The vision module detects the objects in a Zendo scene and predicts their attributes and spatial relations\. Unlike bounding\-box–based detectors, which struggle to represent relations such as touching or pointing, our model outputs a structured symbolic scene description suitable for program\-based rule induction\.

We use a ResNet\-18 backbone followed by a small transformer encoder \(4 layers\) that produces a fixed set of T=7 object tokens\. Each token is decoded by dedicated heads predicting object attributes \(color, shape, orientation\), presence, bounding boxes, and relational properties \(touching and pointing\)\. Relational heads output links between object indices, allowing the model to represent directed and undirected interactions between pieces\. The touching head outputs six values, each representing the direction in which a piece is touching another piece\. The directions are ordered as: left, right, front, back, top, bottom\. This is therefore used for the "TOUCHING" and "ON\_TOP\_OF" predicates in the program synthesis\.

Training uses a permutation\-invariant loss based on the Hungarian algorithm\. Predicted and ground\-truth objects are matched using bounding\-box distance, and losses for attributes, relations, presence, and bounding boxes are computed on the aligned objects\. The final objective is a weighted sum of these components\.

This architecture produces a complete, symbolic scene encoding that feeds directly into the program\-synthesis module\.

### E\.2Program Synthesis

To infer Zendo rules from labeled examples, we use the DeepSynth framework\[[11](https://arxiv.org/html/2607.08233#bib.bib47)\], which supports program induction over a DSL\. The DSL is given in[Sec\. E\.2\.1](https://arxiv.org/html/2607.08233#A5.SS2.SSS1.Px1)which allows Zendo rules to be represented as executable logical programs \. The DSL is directly aligned with the predicates used in data generation, ensuring consistency between the program synthesis component and the Prolog\-based example generator\.

DeepSynth converts this DSL into a probabilistic context\-free grammar \(PCFG\) over rule programs\. In principle, the PCFG probabilities can either be uniform or predicted by neural networks that condition on the input–output examples\. In the uniform setting, all productions are assigned equal weight, and synthesis reduces to enumerating programs by increasing size under a flat prior\. In the learned setting, neural predictors estimate a non\-uniform distribution over productions, biasing search toward plausible hypotheses\.

In the neuro\-symbolic agent reported in the main paper, we use a uniform PCFG without neural predictors\. This choice ensures a more even comparison to VLM\-based agents, which do not have access to our training task distribution, and avoids leaking prior knowledge about Zendo rules into the search prior\. The program synthesis module therefore operates as a purely symbolic enumerative search conditioned only on the observed examples\.

Program induction is performed using heap search, which enumerates candidate programs in decreasing PCFG probability and checks their consistency against all labeled examples\. The search procedure is loss\-optimal with respect to the PCFG and can be instantiated with either uniform or learned probabilities\.

#### E\.2\.1Grammar

##### Types

𝖯𝗂𝖾𝖼𝖾\\displaystyle\\mathsf\{Piece\}::=piece\\displaystyle::=\\texttt\{piece\}𝖲𝗍𝗋𝗎𝖼𝗍𝗎𝗋𝖾\\displaystyle\\mathsf\{Structure\}::=𝖫𝗂𝗌𝗍\(𝖯𝗂𝖾𝖼𝖾\)\\displaystyle::=\\mathsf\{List\}\(\\mathsf\{Piece\}\)𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mathsf\{UnaryPred\}::=𝖯𝗂𝖾𝖼𝖾→𝖡𝗈𝗈𝗅\\displaystyle::=\\mathsf\{Piece\}\\to\\mathsf\{Bool\}𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖯𝗋𝖾𝖽\\displaystyle\\mathsf\{InteractionPred\}::=𝖯𝗂𝖾𝖼𝖾→\(𝖲𝗍𝗋𝗎𝖼𝗍𝗎𝗋𝖾→𝖡𝗈𝗈𝗅\)\\displaystyle::=\\mathsf\{Piece\}\\to\(\\mathsf\{Structure\}\\to\\mathsf\{Bool\}\)𝖱𝗎𝗅𝖾\\displaystyle\\mathsf\{Rule\}::=𝖲𝗍𝗋𝗎𝖼𝗍𝗎𝗋𝖾→𝖡𝗈𝗈𝗅\\displaystyle::=\\mathsf\{Structure\}\\to\\mathsf\{Bool\}𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽::=\\displaystyle\\mathsf\{UnaryPred\}::=~IS\_GROUNDED​∣IS\_UNGROUNDED∣​IS\_RED​∣IS\_BLUE∣​IS\_YELLOW\\displaystyle\\texttt\{IS\\\_GROUNDED\}\\mid\\texttt\{IS\\\_UNGROUNDED\}\\mid\\texttt\{IS\\\_RED\}\\mid\\texttt\{IS\\\_BLUE\}\\mid\\texttt\{IS\\\_YELLOW\}∣IS\_BLOCK∣​IS\_WEDGE​∣IS\_PYRAMID∣​IS\_UPRIGHT\\displaystyle\\mid\\texttt\{IS\\\_BLOCK\}\\mid\\texttt\{IS\\\_WEDGE\}\\mid\\texttt\{IS\\\_PYRAMID\}\\mid\\texttt\{IS\\\_UPRIGHT\}∣IS\_UPSIDE\_DOWN∣​IS\_DOORSTOP∣IS\_CHEESECAKE\\displaystyle\\mid\\texttt\{IS\\\_UPSIDE\\\_DOWN\}\\mid\\texttt\{IS\\\_DOORSTOP\}\\mid\\texttt\{IS\\\_CHEESECAKE\}∣IS\_VERTICAL∣​IS\_FLAT\\displaystyle\\mid\\texttt\{IS\\\_VERTICAL\}\\mid\\texttt\{IS\\\_FLAT\}𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖯𝗋𝖾𝖽::=\\displaystyle\\mathsf\{InteractionPred\}::=~TOUCHING​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽∣POINTING​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\texttt\{TOUCHING\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}\\mid\\texttt\{POINTING\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣ON\_TOP\_OF𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{ON\\\_TOP\\\_OF\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}

##### Syntax

𝖱𝗎𝗅𝖾::=\\displaystyle\\mathsf\{Rule\}::=~𝖴𝗇𝖺𝗋𝗒𝖱𝗎𝗅𝖾​∣𝖡𝗂𝗇𝖺𝗋𝗒𝖱𝗎𝗅𝖾∣​𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖱𝗎𝗅𝖾\\displaystyle\\mathsf\{UnaryRule\}\\mid\\mathsf\{BinaryRule\}\\mid\\mathsf\{InteractionRule\}∣𝖲𝖾𝗍𝖱𝗎𝗅𝖾∣​𝖢𝗈𝗇𝗇𝖾𝖼𝗍𝗂𝗏𝖾𝖱𝗎𝗅𝖾\\displaystyle\\mid\\mathsf\{SetRule\}\\mid\\mathsf\{ConnectiveRule\}𝖴𝗇𝖺𝗋𝗒𝖱𝗎𝗅𝖾::=\\displaystyle\\mathsf\{UnaryRule\}::=~AT\_LEAST\_1​𝖨𝗇𝗍​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\texttt\{AT\\\_LEAST\\\_1\}~\\mathsf\{Int\}~\\mathsf\{UnaryPred\}∣EXACTLY\_1𝖨𝗇𝗍𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EXACTLY\\\_1\}~\\mathsf\{Int\}~\\mathsf\{UnaryPred\}∣ZERO\_1𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{ZERO\\\_1\}~\\mathsf\{UnaryPred\}∣EVEN\_1𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EVEN\\\_1\}~\\mathsf\{UnaryPred\}∣ODD\_1𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{ODD\\\_1\}~\\mathsf\{UnaryPred\}∣EXCLUSIVELY𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EXCLUSIVELY\}~\\mathsf\{UnaryPred\}𝖡𝗂𝗇𝖺𝗋𝗒𝖱𝗎𝗅𝖾::=\\displaystyle\\mathsf\{BinaryRule\}::=~AT\_LEAST\_2​𝖨𝗇𝗍​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽​𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\texttt\{AT\\\_LEAST\\\_2\}~\\mathsf\{Int\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣EXACTLY\_2𝖨𝗇𝗍𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EXACTLY\\\_2\}~\\mathsf\{Int\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣ZERO\_2𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{ZERO\\\_2\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣EVEN\_2𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EVEN\\\_2\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣ODD\_2𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{ODD\\\_2\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣SAME\_AMOUNT𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{SAME\\\_AMOUNT\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}∣MORE\_THAN𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽𝖴𝗇𝖺𝗋𝗒𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{MORE\\\_THAN\}~\\mathsf\{UnaryPred\}~\\mathsf\{UnaryPred\}𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖱𝗎𝗅𝖾::=\\displaystyle\\mathsf\{InteractionRule\}::=~AT\_LEAST\_INTERACTION​𝖨𝗇𝗍​𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖯𝗋𝖾𝖽\\displaystyle\\texttt\{AT\\\_LEAST\\\_INTERACTION\}~\\mathsf\{Int\}~\\mathsf\{InteractionPred\}∣EXACTLY\_INTERACTION𝖨𝗇𝗍𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EXACTLY\\\_INTERACTION\}~\\mathsf\{Int\}~\\mathsf\{InteractionPred\}∣EVEN\_INTERACTION𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{EVEN\\\_INTERACTION\}~\\mathsf\{InteractionPred\}∣ODD\_INTERACTION𝖨𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇𝖯𝗋𝖾𝖽\\displaystyle\\mid\\texttt\{ODD\\\_INTERACTION\}~\\mathsf\{InteractionPred\}𝖲𝖾𝗍𝖱𝗎𝗅𝖾::=\\displaystyle\\mathsf\{SetRule\}::=~EVEN∣ODD\\displaystyle\\texttt\{EVEN\}\\mid\\texttt\{ODD\}∣ALL\_THREE\_SHAPES\\displaystyle\\mid\\texttt\{ALL\\\_THREE\\\_SHAPES\}∣ALL\_THREE\_COLORS\\displaystyle\\mid\\texttt\{ALL\\\_THREE\\\_COLORS\}∣EITHER\_OR𝖨𝗇𝗍𝖨𝗇𝗍\\displaystyle\\mid\\texttt\{EITHER\\\_OR\}~\\mathsf\{Int\}~\\mathsf\{Int\}𝖢𝗈𝗇𝗇𝖾𝖼𝗍𝗂𝗏𝖾𝖱𝗎𝗅𝖾::=\\displaystyle\\mathsf\{ConnectiveRule\}::=~AND​𝖱𝗎𝗅𝖾​𝖱𝗎𝗅𝖾∣OR​𝖱𝗎𝗅𝖾​𝖱𝗎𝗅𝖾\\displaystyle\\texttt\{AND\}~\\mathsf\{Rule\}~\\mathsf\{Rule\}\\mid\\texttt\{OR\}~\\mathsf\{Rule\}~\\mathsf\{Rule\}

## Appendix FAdditional Evaluations

### F\.1Perception Evaluation

We evaluate the vision module on a synthetic test set of 2,779 images\. After Hungarian matching between predicted and ground\-truth objects, the model achieves very high accuracy across all prediction heads, including color \(98\.9%\), shape \(98\.8%\), orientation \(99\.0%\), touching \(98\.4%\), pointing \(96\.9%\), and bounding boxes \(99\.3%\)\. The model also recovers the correct number of pieces in 99\.8% of scenes\.[Table 7](https://arxiv.org/html/2607.08233#A6.T7)summarizes these results\. Qualitative inspection confirms that both object attributes and relations are extracted correctly\.

Table 7:Accuracy and loss summary for each prediction head\.Ablations confirm that transformer layers and dedicated relational heads are needed for generalization: removing transformer layers increases the aggregated validation loss to 1\.76, compared to 0\.0423 for the full model, indicating a substantial degradation in relational prediction\. Using simpler relational heads yields a higher loss of 0\.087\.

## Appendix GHuman Study Platform

![Refer to caption](https://arxiv.org/html/2607.08233v1/figures/frontend_experiment.png)\(a\)Build Scene Screen\.
![Refer to caption](https://arxiv.org/html/2607.08233v1/figures/frontend_label.png)\(b\)Guessing Label Screen\.
![Refer to caption](https://arxiv.org/html/2607.08233v1/figures/frontend_guess.png)\(c\)Guessing Rule Screen\.

Figure 8:Overview of study UI\.### G\.1Participants

We collect data from 19 participants of which five played all six games, two played 4 games, two played three games, six played two games and four played just one game\. Participants were recruited from within the institution\. The participants play time varied between 10 to 40 minutes per game\.

### G\.2System Architecture

The human study is delivered through a purpose\-built web application\. The frontend is a React \+ TypeScript single\-page application served over HTTPS\. The backend is a Python FastAPI service that hosts the game logic and communicates with the frontend exclusively via WebSocket\. The backend reuses the same GameMaster and DSL infrastructure as the agent experiments, ensuring that the rules, counterexample generation, and labeling logic are identical for human and AI players\. The backend system is run with Docker and deployed on a internal university server and the frontend is deployed on a Jatos server\.

### G\.3Study Procedure

Participants access the study through a JATOS URL\. The study proceeds in three phases:

1. 1\.Instructions and consent\.Participants read a study information sheet explaining the purpose, procedure, expected duration \(≈\\approx45 minutes\), data collected, storage, and their right to withdraw at any time\. A required checkbox confirms informed consent before the study begins\.
2. 2\.Tutorial\.Participants complete one full guided game in which on\-screen annotations explain the interface elements and game rules\. Tutorial data are excluded from analysis\.
3. 3\.Study games\.Participants play a subset of the 22ZendoWorldepisodes following the same protocol as the AI Agents of the main paper\.[Figure 8](https://arxiv.org/html/2607.08233#A7.F8)shows the three screens the user sees during the game\. The first image \([8\(a\)](https://arxiv.org/html/2607.08233#A7.F8.sf1)\) is during the experimentation phase where the user can build the scene, the second image \([8\(b\)](https://arxiv.org/html/2607.08233#A7.F8.sf2)\) shows the user the scene and asks for a label and the third image \([8\(c\)](https://arxiv.org/html/2607.08233#A7.F8.sf3)\) shows the rule guessing screen where the user can input their hypothesis\.

### G\.4Data Collection and Storage

Two complementary log files are produced per episode and stored on university servers\.

##### Action log\.

A fine\-grained event stream capturing every user interaction during scene construction and decision\-making\. Each entry contains an ISO 8601 timestamp and an action type together with action\-specific details\. Logged events include:

- •Scene building:piece\_added,piece\_orientation\_cycled,piece\_moved,piece\_touching\_set/removed,piece\_pointing\_set/removed,piece\_stacked\_set/removed\.
- •Guessing:guess\_label\(YES/NO prediction\),quiz\_result\(correctness and stone count\),rule\_typed\(full text of hypothesis\),rule\_submitted,rule\_skipped\.
- •Navigation:screen\_change,received\_labeled\_example,game\_started,game\_over,session\_complete\.

##### Game state file\.

A structured JSON summary saved at episode completion containing: the ground\-truth DSL rule, rule complexity, number of turns and examples, all rule hypotheses submitted, label\-guess correctness per turn, per\-turn durations in seconds, the game\-over reason, and file paths to the rendered scene images\.

##### Scene images\.

For every scene shown to or proposed by a participant, a640×480640\\times 480PNG rendered by Blender is saved alongside the game state file\. Images are named by source \(gmfor Game\-Master\-provided,playerfor participant\-proposed\) and ground\-truth label\.

##### JATOS submission\.

Upon episode completion the frontend submits a result payload to the JATOS server viajatos\.submitResultData\. The payload contains the session metadata \(anonymous participant ID, browser user\-agent, screen resolution\) and the complete action log for each completed episode\.

### G\.5Anonymisation and Privacy

No personal data are collected at any point\. Each participant is assigned an anonymous session ID of the forms\_XXXXXXXX\(8\-character hex string\) generated client\-side and stored in browserlocalStorage\. All stored files and JATOS result entries are keyed by this ID\. If JATOS is available, the JATOS worker ID is also recorded to enable session linkage within the JATOS platform, but this ID carries no personally identifiable information\. No IP addresses or identifying metadata are logged by the application\. The privacy notice presented to participants states explicitly that the data cannot be traced back to any individual\.

### G\.6Participant Compensation and Voluntariness

Participation is entirely voluntary and participants may withdraw at any time by closing the browser window without any negative consequences\. No compensation is provided\. Data from sessions that do not reach the study games phase are excluded from analysis\.

### G\.7Instructions

The following instructions are given to the participants before starting the games:

Information Sheet and Data Protection

Purpose and Potential Benefit\.The purpose of this study is to examine how people learn and reason about visual rules\. The results have the potential to improve our understanding of human inductive reasoning and concept learning\.

Procedure\.After a short briefing and tutorial, participants will play a computer\-based game\.

Duration\.Participation takes approximately 20 minutes\. There is no monetary compensation\.

Risks\.Participation does not involve any risks beyond those encountered in everyday life\.

Privacy\.No personal data is collected\. We record only task\-related interactions, including constructed scenes, label guesses, rule guesses, and timing information\. These data are linked to an anonymous participant ID and cannot be traced back to individuals\.

Storage\.Data are stored on university servers and internal systems in anonymised form and may be used in scientific publications\.

Voluntariness\.Participation is voluntary\. Participants may withdraw at any time without disadvantage\.

Consent\.By continuing, participants agree to participate and consent to anonymous data storage and use\.

Contact\.Name \(email\)\.

Game Instructions: Zendo

Goal\.Identify a hidden rule inas few turns as possible\.

Pieces\.Scenes consist of 3D pieces varying in:

- •Shape: block, pyramid, wedge
- •Color: red, blue, yellow
- •Orientation: upright, upside\-down, flat, cheesecake, doorstop

Pieces may touch, stack, or point toward each other\.

Start of the Game\.You are shown one positive \(YES\) and one negative \(NO\) example\.

Turn Structure\.

1. 1\.Build a scene to test your hypothesis\.
2. 2\.Guess whether it follows the rule \(YES/NO\)\.
3. 3\.If correct, optionally guess the rule; otherwise a counter\-example is shown\.

Ending\.The game ends when the rule is found or after 30 labelled scenes\.

Tips\.

- •Compare YES and NO scenes\.
- •Change one feature at a time\.
- •Track previous guesses\.

## Appendix HHuman Study Results

The results of the human study are reported in[Table 8](https://arxiv.org/html/2607.08233#A8.T8)combined with the Agent results per task\.

Table 8:Per\-task results on the 6 human\-study tasks\. Solved shows wins/total \(seeds for agents, participants for humans\)\. Avg Trn shows mean turns±\\pmSEM for winning runs only across seeds\. – = no winning runs\.
## Appendix IAgents

### Oracle Agent

The Oracle Agent perceives each Zendo scene by passing its rendered image through a pre\-trained ZendoImageToVector\-Model, a transformer\-based object encoder that produces a 7×15 integer tensor, encoding each piece’s object ID, color, shape, orientation, six touching\-relation slots, and pointing target\. To propose a new experiment, the agent first runs a bigram\-prior PCFG heap search over the hand\-crafted Zendo DSL which is a typed library of predicates such as IS\_RED, AT\_LEAST, ODD, EXACTLY, TOUCHING, and POINTING, enumerating programs in descending prior\-probability order while filtering known\-incorrect and canonically equivalent programs\. The bigram\-prior PCFG is a probabilistic context\-free grammar over the DSL whose production probabilities are conditioned on the parent rule and argument index, with the conditional weights being uniformly distributed\. If no fully consistent program is found, the accuracy threshold is relaxed by one example at a time until a candidate survives\. The top DSL program is then compiled back to Prolog to function calls\. The primary proposal strategy calls the Prolog engine with a query including both Prolog function calls, generating a scene that either satisfiesr​u​l​e2rule\_\{2\}but notr​u​l​e1rule\_\{1\}or the other way around\. If only one candidate exists, it falls back to generate an example that either aligns with the rule or not\. The resulting Prolog scene string is rendered to a PNG via Blender\. To label an externally proposed scene, the agent evaluates the top\-ranked DSL program on its tensor encoding\. To guess the rule, it returns the highest\-scoring program from the heap search\.

### VLM Agent

The VLM Agent delegates all perception, generation, and inference entirely to a multimodal large language model \(GPT\-5\-mini\), with no symbolic program search\. Each observed example is stored as a raw \(tensor, label\) pair alongside its rendered image path, the tensor is computed once from the Prolog scene string but is not used for reasoning\. To propose a new experiment, the agent encodes all positive and negative example images as base64 data URIs, constructs a multimodal prompt interleaving these images with a task description, and instructs the LLM to output a Python literal of the form \[\[item\(id, color, shape, orientation, grounding\), …\], label\]\. The proposed label is cached and returned verbatim when the game engine subsequently asks the agent to classify the scene it just proposed, avoiding a redundant LLM call\. To guess the rule, the agent assembles an analogous multimodal prompt from all labeled example images and requests a natural\-language rule\. The response is returned as\-is without compilation or consistency verification against the examples\.

### Bayesian Agent

The Bayesian Agent maintains a particle approximation to a posterior over natural\-language hypotheses, combining LLM\-based hypothesis generation with probabilistic filtering over concept descriptions\. Each hypothesis is a free\-form English rule \(e\.g\., “there must be a red block touching a pyramid”\), which is evaluated by first translating it into executable code using the rule\-conversion prompt listed in[App\. J](https://arxiv.org/html/2607.08233#A10)\. The resulting program is then executed on a scene to determine whether the hypothesis predicts a positive or negative label\.

On the first turn, the agent selects a random positive example and, for each of seven attribute families \(colors, shapes, orientations, groundedness, touching, pointing, and stacking\), promptsgpt\-5\-minito generate five candidate hypotheses conditioned on that image\. These candidates form the initial particle set\. Given a labeled scene, each particle is scored by its marginal likelihood under a noisy concept\-learning model with recall parameterθ\\thetaand specificity parameterδ\\delta\. We place Gaussian priors onθ\\thetaandδ\\delta, withθ∼𝒩​\(0\.7,0\.1\)\\theta\\sim\\mathcal\{N\}\(0\.7,0\.1\)andδ∼𝒩​\(0\.9,0\.01\)\\delta\\sim\\mathcal\{N\}\(0\.9,0\.01\), truncated to\[0\.5,1\.0\]\[0\.5,1\.0\]\. For a scenexxwith labelyy, the likelihood is high when the translated hypothesis predictsyyand lower otherwise, with false positives and false negatives weighted asymmetrically according toθ\\thetaandδ\\delta\. The final particle weight is obtained by marginalizing this likelihood over the\(θ,δ\)\(\\theta,\\delta\)prior grid\.

Particles are resampled to a support of 25 hypotheses using systematic resampling\. After each new observation, the lowest\-likelihood particles are rejuvenated by prompting the LLM with the latest labeled scene and the current hypothesis to propose refined variants; only refinements that improve the joint score are retained\. To propose a new experiment, the agent estimates the expected KL divergence between the posterior before and after observing the outcome of each candidate scene, and selects the scene with the highest expected information gain\. Candidate scenes are generated by prompting the LLM with the current high\-weight hypotheses to produce Prolog\-style scene descriptions for each label\. To label a scene, the agent marginalizesP​\(yes∣x,c\)P\(\\mathrm\{yes\}\\mid x,c\)over the particle support\. To guess the rule, it returns the MAP particle, i\.e\., the most frequent hypothesis after resampling\.

### VLP Agent

The Visual Language Programming Agent induces a concept vocabulary from images before searching for rules\. Upon each of the first 5 observations, the agent queries a vision\-language prompter with the newly received image to discover novel object categories \(target: 6\), visual properties \(target: 9\), and spatial actions/relations \(target: 5\) not yet in its vocabulary\. Each newly identified concept string is registered as a typed DSL primitive \(objects as OBJECT, properties as PROPERTY, relations as ACTION\) dynamically extending the grammar\. Subsequent rounds include a "Already Discovered" header so the VLM only returns genuinely new vocabulary\. Program search then runs a bigram\-prior PCFG heap search over this evolving VLP DSL\. During evaluation a VLM is prompted using raw image paths rather than fixed tensors to detect pieces in images, letting the language model act as the semantic interpreter\. To propose a new experiment, the agent runs the VLP search, takes the top two surviving programs, and constructs a multimodal prompt with all labeled example images and the candidate rules\. The VLM responds with a \[\[item\(…\), …\], label\] Prolog literal that is parsed, validated against the 7×15 tensor schema, and rendered via Blender\. To label a scene or guess a rule, the agent returns the result of the top candidate, evaluated on the image or the string form of the top\-ranked VLP program respectively\.

## Appendix JAgent Prompts

During the hypothesis evaluation of the agents, we use the following prompt to convert the natural language hypotheses into processable DSL strings:

Convertthefollowinghypothesis,fromadifferentDSLaboutZendostructuresintoaDSLrepresentationusingtheprovidedsyntax\.

OutputonlytheDSLexpressionwithoutanyadditionaltext\.IftherulecannotberepresentedwiththeDSL,returnsomethingthatissimilar\.

Hypothesis:\{hs\}

DSLSyntax:

UseS\-expressions:\(OParg1arg2\.\.\.\)withprefixnotation\.

AcompleteoutputmustbeasingleDSLexpression\.

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Unarypredicates\(noarguments\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Colors:

\-IS\_RED

\-IS\_BLUE

\-IS\_YELLOW

Shapes:

\-IS\_BLOCK

\-IS\_WEDGE

\-IS\_PYRAMID

Grounding:

\-IS\_GROUNDED

\-IS\_UNGROUNDED

Orientation:

\-IS\_UPRIGHT

\-IS\_UPSIDE\_DOWN

\-IS\_VERTICAL

\-IS\_FLAT

\-IS\_DOORSTOP

\-IS\_CHEESECAKE

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Interactionpredicates

\(take2unarypredicates;returnaninteractionpredicate\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(TOUCHINGP1P2\)

\-\(ON\_TOP\_OFP1P2\)

\-\(POINTINGP1P2\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Logicalrulecombinators

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(ANDR1R2\)

\-\(ORR1R2\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Globalrules\(noarguments\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(ALL\_THREE\_COLORS\)

\-\(ALL\_THREE\_SHAPES\)

\-\(EVEN\);totalnumberofpiecesiseven

\-\(ODD\);totalnumberofpiecesisodd

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Countrulesoverunarypredicates

\(INTnin\(1,2,3,4,5,6,7\)\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(AT\_LEAST\_1nP\)

\-\(EXACTLY\_1nP\)

\-\(ZERO\_1P\)

\-\(EVEN\_1P\);nonzeroevencount

\-\(ODD\_1P\);oddcount

\-\(EXCLUSIVELYP\);allpiecessatisfyP

\-\(ALL\_1P\);allpiecessatisfyP

\-\(MAJORITY\_1P\);atleasthalfofpiecessatisfyP

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Countrulesoverconjunctiononthesamepiece

\(INTnin\(1,2,3,4,5,6,7\)\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(AT\_LEAST\_2nP1P2\)

\-\(EXACTLY\_2nP1P2\)

\-\(ZERO\_2P1P2\)

\-\(EVEN\_2P1P2\)

\-\(ODD\_2P1P2\)

\-\(ALL\_2P1P2\);allpiecessatisfy\(P1ANDP2\)

\-\(MAJORITY\_2P1P2\);atleasthalfsatisfy\(P1ANDP2\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Countrulesoverinteractions

\(INTnin\(1,2,3,4,5,6,7\)\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(AT\_LEAST\_INTERACTIONn\(RELP1P2\)\)

\-\(EXACTLY\_INTERACTIONn\(RELP1P2\)\)

\-\(EVEN\_INTERACTION\(RELP1P2\)\)

\-\(ODD\_INTERACTION\(RELP1P2\)\)

\-\(MAJORITY\_INTERACTION\(RELP1P2\)\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Othernumeric/comparisonrules

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

\-\(LENGTHn\);totalnumberofpiecesisexactlyn

\-\(EITHER\_ORn1n2\);totalnumberofpiecesisn1orn2

\-\(MORE\_THANP1P2\);count\(P1\)\>count\(P2\)

\-\(MORE\_OR\_EQUAL\_THANP1P2\);count\(P1\)\>=count\(P2\)

\-\(SAME\_AMOUNTP1P2\);count\(P1\)==count\(P2\)and\>0

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Examples/templates

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Singleattribute:

\-"atleast3wedges":

\(AT\_LEAST\_13IS\_WEDGE\)

Twoattributesonsamepiece:

\-"exactly2redblocks":

\(EXACTLY\_22IS\_REDIS\_BLOCK\)

Interactioncount:

\-"atleastoneblueontopofayellow":

\(AT\_LEAST\_INTERACTION1\(ON\_TOP\_OFIS\_BLUEIS\_YELLOW\)\)

Majority:

\-"mostpiecesarered":

\(MAJORITY\_1IS\_RED\)

Comparisons:

\-"moreredpiecesthanbluepieces":

\(MORE\_THANIS\_REDIS\_BLUE\)

Combineclauses:

\-\(ANDR1R2\)

\-\(ORR1R2\)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

ReturnonlytheDSLexpressionbelowinsideasinglecodeblock,likethis:

‘‘‘python

\(\.\.\.\)

‘‘‘

If this fails we use the following prompt to ask for equivalence:

WeareplayingthegameZendowiththefollowingattributesforthepieces:

\-color:red,blue,yellow;

\-shape:block,wedge,pyramid;

\-orientation:upright,upside\_down,flat,cheesecake,doorstop;

\-relation:grounded\(whetherthepieceistouchingtheground\),touching\(ID\)\(whetherthepieceistouchinganotherpiecewithID\),pointing\(ID\)\(whetherthepieceispointingtoanotherpiecewithID\),on\_top\_of\(ID\)\(whetherthepieceisontopofanotherpiecewithID\)\.

Wedgesareneverflatbutinsteadcanbedoorstoporcheesecake,whilethetwoothershapescanbeflatbutnotcheesecakeordoorstop\.

Interactionscanbe:grounded,touching\(ID\),pointing\(ID\)andon\_top\_of\(ID\),whereIDisthefirstfieldofanotherpiece,e\.g\."pointing\(2\)"meansthispieceispointingtothepiecewithID2\.

IsthefirstDSLrulesemanticallyequivalenttothesecondDSLrule?

Rule1:

\{rule1\}

Rule2:

\{rule2\}

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Outputformat

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

YouMUSTreturnexactlyoneofthefollowing:

Case1\-\-Equivalent:

‘‘‘json

\{\{"equivalent":true\}\}

‘‘‘

Case2\-\-Notequivalent:

‘‘‘json

\{\{

"equivalent":false,

"counterexample":"<structuredescription\>"

\}\}

‘‘‘

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Counterexamplerequirements

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Thecounterexampleevaluatedifferentlyonthetworules\.

DescribethestructureusingONLYthisformat:

piece1:color=\.\.\.,shape=\.\.\.,orientation=\.\.\.,relation=\.\.\.

piece2:color=\.\.\.,shape=\.\.\.,orientation=\.\.\.,relation=\.\.\.

\.\.\.

DoNOTincludeexplanations

DoNOTincludereasoning

DoNOTincludeanytextoutsidetheJSON

ReturnONLYtheJSONblock\.

Using the output from the equivalence check, we ask the LLM for a counter example using the following prompt:

Astructurehasoneormorepieces/items\.Eachpieceshouldcontainthefollowingattributes:

\-color:red,blue,yellow;

\-shape:block,wedge,pyramid;

\-orientation:upright,upside\_down,flat,cheesecake,doorstop;

\-relation:grounded\(whetherthepieceistouchingtheground\),touching\(ID\)\(whetherthepieceistouchinganotherpiecewithID\),pointing\(ID\)\(whetherthepieceispointingtoanotherpiecewithID\),on\_top\_of\(ID\)\(whetherthepieceisontopofanotherpiecewithID\)\.

Wedgesareneverflatbutinsteadcanbedoorstoporcheesecake,whilethetwoothershapescanbeflatbutnotcheesecakeordoorstop\.

Interactionscanbe:grounded,touching\(ID\),pointing\(ID\)andon\_top\_of\(ID\),whereIDisthefirstfieldofanotherpiece,e\.g\."pointing\(2\)"meansthispieceispointingtothepiecewithID2\.

Yourtaskistoconvertthefollowingstructuredescriptionintothewantedformat\.

Description:\{description\}

ONLYreturnanewexamplewithinapythonblock,inthisexactformat\.

Hereareexamplesofvalidformats:

\["item\(0,red,block,upright,grounded\)","item\(1,blue,wedge,doorstop,touching\(0\)\)"\]

\["item\(0,yellow,pyramid,upside\_down,grounded\)","item\(1,blue,wedge,doorstop,on\_top\_of\(0\)\)","item\(2,red,block,upright,pointing\(1\)\)"\]

Returnyouranswerinthisexactformatincludingthepythonblock:

‘‘‘python

\["item\(ID,color,shape,orientation,interaction\)",\.\.\.\]

‘‘‘

### J\.1VLM Agent

The VLM Agent uses the following promt to propose examples:

YouareaZendoplayer\.Yourgoalisto\*\*gainnewinformation\*\*aboutthehiddenrulebyproposinga

\*\*novel\*\*structurethatis\*\*maximallyinformative\*\*\(highlylikelytochangeorconfirmcurrentbeliefs\)\.

Youaregiven\*visual\*positiveandnegativeexamples\.Studytheimages,butoutputyourproposalas\*\*text\*\*\.

Thepiecescanhave

\-colors:red,blue,yellow;

\-shapes:block,wedge,pyramid;

\-orientations:upright,upside\_down,flat,cheesecake,doorstop\.

Wedgesareneverflatbutinsteadcanbedoorstoporcheesecake,whilethetwoothershapescanbeflatbutnotcheesecakeordoorstop\.

Interactionscanbe:grounded,touching\(ID\),pointing\(ID\)andon\_top\_of\(ID\),whereIDisthefirstfieldofanotherpiece,e\.g\."pointing\(2\)"meansthispieceispointingtothepiecewithID2\.

DONOTincludeexplanations,reasoning,comments,ortextoutsidethepythonblock,inthisexactformat,wherelabelis1forvalidand0forinvalid:

Hereareexamplesofvalidanswers:

‘‘‘python

\[\["item\(0,red,block,upright,grounded\)","item\(1,blue,wedge,doorstop,touching\(0\)\)"\],1\]

‘‘‘

‘‘‘python

\[\["item\(0,yellow,pyramid,upside\_down,grounded\)","item\(1,blue,wedge,doorstop,on\_top\_of\(0\)\)","item\(2,red,block,upright,pointing\(1\)\)"\],0\]

‘‘‘

Returnyouranswerinthisexactformatincludingthepythonblock:

‘‘‘python

\[\["item\(ID,color,shape,orientation,interaction\)",\.\.\.\],label\]

To guess rules, the Agent uses the following prompt:

YouareaZendoplayer\.Yourjobistofindanewlogicalclassificationruleforgivenexampleswithlabels\.Youaregivenafewpositiveandnegativeexamples\.Eachimageconsistsofpiecesindifferentconfigurations\.

\*\*Availablevalues:\*\*

\-Colors:red,blue,yellow

\-Shapes:block,wedge,pyramid

\-Orientations:upright,upside\_down,flat,cheesecake,doorstop,vertical

\-Interactions:grounded,touching,pointing,on\_top\_of,groundedness

\*\*Goal:\*\*

Find\*\*one\*\*rulethatis\*\*Trueforallpositiveexamples\*\*and\*\*Falseforallnegativeexamples\*\*\.

\*\*Criticalrulesforselectingyouranswer:\*\*

1\.Youmaycombineshortrulesusing"and"or"or"ifneeded,but\*\*ifasinglepredicateworks,useit\.\*\*

2\.Therulemustbeasshortandsimpleaspossiblebutstillaccurate\.

3\.Return\*\*only\*\*therule\-noexplanation,noformatting,noextratext\.

4\.Donotuseanyconditionals\("if","when","onlyif",etc\.\)oranytextoutsidetheruleitself\.

\#\#\#OutputFormat:

Return\*\*only\*\*asingleruleinnaturallanguage\.Do\*\*not\*\*includeexplanationsorextratext\.

### J\.2VLP Agent

The following prompts were used in the beginning of the game to discover predicates over the images: Object Discovery:

Youareanalyzingimagestoidentifythebaseshapesofobjectsthatappear\.Focusonthefundamentalshapeorkindofeachobject,notitscolor,size,ororientation\-thoseareproperties,notobjecttypes\.

IMPORTANT:

\-Returnonlyatomicobjecttypenames\(e\.g\.,"block","wedge","pyramid","sphere","cube","doorstop"\)

\-DoNOTincludecolor,size,ororientationintheobjectname\(e\.g\.,"redblock"iswrong,use"block"\)

\-DoNOTcombinemultipleattributesintoonename\(e\.g\.,"yellowcheesecake"iswrong,use"cheesecake"\)

\-Ifthesameshapeappearsindifferentcolorsorsizes,itisstillthesameobjecttype

\-Returnexactly\{n\}objectsinaPythonlist

Answerformat:

‘‘‘python

objects=\[\.\.\.\]

‘‘‘

Nocommentsorexplanations\.Ifnoobjectsfound,return\[\]\.

Property Discovery:

\#PropertyDiscoveryTask

Youareanalyzingasetofimagestoidentifyimportantpropertiesthatdescribeobjectsintheimageset\.

\#\#Objective

Discover\*\*atomicvisualproperties\*\*thatcharacterizeobjectsintheimages\.Eachpropertyshouldbeasingleattribute,notacombination\.

\#\#Instructions

1\.\*\*Examineallimagescarefully\*\*\-Lookforpropertiesthatapplytotherelevantobjectsacrosstheimageset

2\.\*\*Identifyimportantproperties\*\*\-Focusonsignificant,clearlyobservablepropertiesthatmeaningfullydescribeobjects

3\.\*\*Considerpropertyvariation\*\*\-Propertiesthatvaryacrossimagesorobjectsmaybeparticularlynoteworthy

4\.\*\*Prioritizemeaningfulproperties\*\*\-Choosepropertiesthathelpdistinguishorcharacterizeobjects\(e\.g\.,color,size,position,orientation,state\)

5\.\*\*Returnexactly\{n\}properties\*\*\-Iffewernotablepropertiesexist,returnasmanyasavailable

6\.\*\*Usedescriptivenames\*\*\-Namepropertiesclearlyandspecifically\(e\.g\.,"red"ratherthan"colored","horizontal"ratherthan"oriented"\)

\#\#RelevantObjects

Theobjectstoconsiderare:\{objects\}

\#\#PropertyCategories

\-\*\*Visualattributes\*\*:colors\(red,green,blue,yellowetc\.\)

\-\*\*Geometricattributes\*\*:orientations\(upright,flat,upside\_downetc\.\)

\-\*\*Groundedness\*\*:grounded,ungrounded

\#\#OutputRequirements

\-ReturnaPythonlistassignedtovariable‘properties‘

\-IncludeonlythePythoncode,noexplanationsorcomments

\-Ifnonotablepropertiesarefound,returnanemptylist‘\[\]‘

\-Usesingle\-wordorsimplepropertynames\(e\.g\.,"blue","upright"\)

\-DoNOTincludeobjecttypesasproperties\(e\.g\.,"wedge"\)

\#\#ExampleFormat

‘‘‘python

properties=\["red","blue","yellow","upright","flat","doorstop","grounded","small"\]

‘‘‘

Action Discovery:

Youareanalyzingasetofimagestoidentifyimportantrelationsbetweenobjects\.

Discover\*\*notablerelations\*\*thatcharacterizetheplacementoftwoobjectsintheimages\.Focusonrelationsthatmeaningfullydistinguishimages\.

1\.\*\*Examineallimagescarefully\*\*\-Lookforrelationsthatapplytotherelevantobjectsacrosstheimageset

2\.\*\*Identifyimportantrelations\*\*\-Focusonsignificant,clearlyobservableactionsthatmeaningfullydescribewhatobjectsaredoing

3\.\*\*Considerrelationvariation\*\*\-Relationsthatvaryacrossimagesorobjectsmaybeparticularlynoteworthy\(contrastingactions\)

4\.\*\*Returnexactly\{n\}relations\*\*\-Iffewernotablerelationsexist,returnasmanyasavailable

5\.\*\*Usedescriptivenames\*\*\-Namerelationsclearlyandspecifically\(e\.g\.,"running"ratherthan"moving","sitting"ratherthan"positioned","touching"ratherthan"close"\)

\#\#RelevantObjects

Theobjectstoconsiderare:\{objects\}

\#\#OutputRequirements

\-ReturnaPythonlistassignedtovariable‘actions‘

\-IncludeonlythePythoncode,noexplanationsorcomments

\-Ifnonotableactionsarefound,returnanemptylist‘\[\]‘

\-Useclear,specificactionnames

\#\#ExampleFormat

‘‘‘python

actions=\["touching","pointing\_to"\]

‘‘‘

The following prompts are used within the DSL programs evaluation to detect pieces with specific variables in the images: Object detection:

\#\#Task

Identifyobjectsandtheirpropertiesfromtheimageusingonlytheprovidedlists\.

\*\*Objects:\*\*\{objects\}

\*\*Properties:\*\*\{properties\}

\#\#Rules

1\.Onlyuseobjects/propertiesfromtheprovidedlists

2\.Returnemptylistifnovalidobjectsfound

3\.Noexplanationsoradditionaltext

\#\#OutputFormat

‘‘‘python

objects=\[

\[’object\_name’,’property1’,’property2’,\.\.\.\],

\[’object\_name’,’property1’\],

\.\.\.

\]

‘‘‘

\*\*Ifnovalidobjects:\*\*‘objects=\[\[\]\]‘

\#\#Examples

\*\*Example1\*\*

\-Objects:\["car","person","tree"\]

\-Properties:\["red","tall","small","standing"\]

\-Image:Redcarundertalltreewithsmallstandingperson

‘‘‘python

objects=\[

\[’car’,’red’\],

\[’tree’,’tall’\],

\[’person’,’standing’,’small’\]

\]

‘‘‘

\*\*Example2\*\*

\-Objects:\["dog","ball","book","chair"\]

\-Properties:\["blue","sitting","round"\]

\-Image:Dogsittingbyroundballandbluechair

‘‘‘python

objects=\[

\[’dog’,’sitting’\],

\[’ball’,’round’\],

\[’chair’,’blue’\]

\]

‘‘‘

\*\*Example3\*\*

\-Objects:\["bicycle","lamp","table","cup"\]

\-Properties:\["green","broken","wooden","white"\]

\-Image:Tablewithlaptopandcup

‘‘‘python

objects=\[\[\]\]

‘‘‘

\*Note:Eventhough’table’and’cup’areintheobjectslistandvisibleintheimage,neitherhaspropertiesfromtheprovidedlist,sonovalidobject\-propertycombinationsexist\*

\*\*Analyzetheimagenow:\*\*

Property detection:

\#\#Task

Identifyandextractpropertiesforeachobjectintheimageusingonlytheprovidedlists\.

\*\*Properties:\*\*\{properties\}

\#\#Rules

1\.Onlyusepropertiesfromtheprovidedlists

2\.Returnemptylistifnovalidobjectsfound

3\.Noexplanationsoradditionaltext

\#\#OutputFormat

‘‘‘python

properties=\[

\[’property1’,’property2’,\.\.\.\],

\[’property1’\],

\.\.\.

\]

‘‘‘

\*\*Ifnovalidobjects:\*\*‘objects=\[\[\]\]‘

\#\#Examples

\*\*Example1\*\*

\-Objects:\["car","person","tree"\]

\-Properties:\["red","tall","small","standing"\]

\-Image:Redcarundertalltreewithsmallstandingperson

‘‘‘python

properties=\[

\[’red’\],

\[’tall’\],

\[’standing’,’small’\]

\]

‘‘‘

\*\*Example2\*\*

\-Objects:\["dog","ball","book","chair"\]

\-Properties:\["blue","sitting","round"\]

\-Image:Dogsittingbyroundballandbluechair

‘‘‘python

properties=\[

\[’sitting’\],

\[’round’\],

\[’blue’\]

\]

‘‘‘

\*\*Example3\*\*

\-Properties:\["green","broken","wooden","white"\]

\-Image:Tablewithlaptopandcup

‘‘‘python

properties=\[\[\]\]

‘‘‘

\*Note:Eventhough’table’and’cup’areintheobjectslistandvisibleintheimage,neitherhaspropertiesfromtheprovidedlist,sonovalidobject\-propertycombinationsexist\*

\*\*Analyzetheimagenow:\*\*

Action detection:

\#\#Task

Identifyinteractiontriplesintheimageusingonlytheprovidedlists\.

Eachtriplehastheformat\[subject,action,object\]where:

\-subject:theentityperformingtheaction\(fromObjectsorProperties\)

\-action:whatishappeningbetweenthem\(fromActions\)

\-object:theentitytheactionisdirectedat\(fromObjectsorProperties\)

\*\*Objects:\*\*\{objects\}

\*\*Properties:\*\*\{properties\}

\*\*Actions:\*\*\{actions\}

\#\#Rules

1\.Onlyusevaluesfromtheprovidedlistsforeachposition

2\.Eachtriplemusthaveexactly3elements:\[subject,action,object\]

3\.Ifthesameinteractionoccursmultipletimesintheimage,includeoneentryperoccurrence

4\.Returnemptylistifnovalidtriplesfound

5\.Noexplanationsoradditionaltext

\#\#OutputFormat

‘‘‘python

actions=\[

\[’subject1’,’action1’,’object1’\],

\[’subject2’,’action2’,’object2’\],

\.\.\.

\]

‘‘‘

\*\*Ifnovalidinteractions:\*\*‘actions=\[\[\]\]‘

\#\#Examples

\*\*Example1\*\*

\-Objects:\["block","pyramid","wedge"\]

\-Properties:\["blue","red"\]

\-Actions:\["touching","grounded"\]

\-Image:Twoblueblockseachtouchingapyramid

‘‘‘python

actions=\[

\[’block’,’touching’,’pyramid’\],

\[’block’,’touching’,’pyramid’\]

\]

‘‘‘

\*Note:Theinteractionoccurstwice,soitappearstwiceinthelist\*

\*\*Example2\*\*

\-Objects:\["block","wedge"\]

\-Properties:\["blue","red","upright"\]

\-Actions:\["touching","supporting"\]

\-Image:Blueblocktouchingredwedge,wedgesupportingablock

‘‘‘python

actions=\[

\[’blue’,’touching’,’red’\],

\[’wedge’,’supporting’,’block’\]

\]

‘‘‘

\*\*Analyzetheimagenow:\*\*

The following prompt is used during the experimentation to get an example suggestion from the VLM:

YouareaZendoplayer\.Yourjobistogenerateanewstructureexampletogainnewknowledgeaboutthehiddenrule\.

Youaregivenafewpositiveandnegativeexamples\.Eachstructureconsistsofalistofitemswiththeformat:

"item\(ID,color,shape,orientation,interaction\)"\.

Thepiecescanhavecolors:red,blue,yellow;shapes:block,wedge,pyramid;orientations:upright,upside\_down,flat,cheesecake,doorstop\.

Wedgesareneverflatbutinsteadcanbedoorstoporcheesecake,whilethetwoothershapescanbeflatbutnotcheesecakeordoorstop\.

Interactionscanbe:grounded,touching\(ID\),pointing\(ID\)andon\_top\_of\(ID\),whereIDisthefirstfieldofanotherpiece,e\.g\."pointing\(2\)"meansthispieceispointingtothepiecewithID2\.

Youmayproposeupto7piecesinyourstructure\.

Thecurrenttoprulehypothesesare:

\{top\_rules\_str\}

Positiveexamples:First\{positive\_count\}examplesarepositive,meaningtheyfollowthehiddenrule,whilethenext\{negative\_count\}examplesarenegative,meaningtheydonotfollowthehiddenrule\.

Hereareexamplesofvalidanswers:

‘‘‘python

\[\["item\(0,red,block,upright,grounded\)","item\(1,blue,wedge,doorstop,touching\(0\)\)"\],1\]

‘‘‘

‘‘‘python

\[\["item\(0,yellow,pyramid,upside\_down,grounded\)","item\(1,blue,wedge,doorstop,on\_top\_of\(0\)\)","item\(2,red,block,upright,pointing\(0\)\)"\],0\]

‘‘‘

PleaseONLYreturnanewexampleanditslabelwithinapythonblock,inthisexactformat,wherelabelis1forvalidand0forinvalid:

‘‘‘python

\[\["item\(ID,color,shape,orientation,interaction\)",\.\.\.\],label\]

‘‘‘

### J\.3Bayesian Agent

The Bayesian Agent uses the following prompt to get initial hypotheses:

Pleaselist\{num\}possiblerulesaboutthe\{att\}withthechoices\{att\_choices\}\.

Example1:

\{example\}

DoNOTproposerulescontainingnonesuchas"thereisawedgepointingatnothing/none"\.

Task1:

Imageisattached\.

Simplerules\(OrdersdoNOTmatter\):

During the experimentation, we use the following prompt to get examples:

Giventherule’\{h\}’,pleasegiveonestructurethatconformswiththeruleandanotherstructurethatviolateswiththerule\.

Astructurehasoneormorepieces\.Eachpieceshouldcontainthefollowingattributes:

\-color:red,blue,yellow;

\-shape:block,wedge,pyramid;

\-orientation:upright,upside\_down,flat,cheesecake,doorstop;

\-relation:grounded\(whetherthepieceistouchingtheground\),touching\(ID\)\(whetherthepieceistouchinganotherpiecewithID\),pointing\(ID\)\(whetherthepieceispointingtoanotherpiecewithID\),on\_top\_of\(ID\)\(whetherthepieceisontopofanotherpiecewithID\)\.

Wedgesareneverflatbutinsteadcanbedoorstoporcheesecake,whilethetwoothershapescanbeflatbutnotcheesecakeordoorstop\.

Interactionscanbe:grounded,touching\(ID\),pointing\(ID\)andon\_top\_of\(ID\),whereIDisthefirstfieldofanotherpiece,e\.g\."pointing\(2\)"meansthispieceispointingtothepiecewithID2\.

Useprologstylelistsofitemstorepresentstructures\.Eachstructureshouldbelabeledwith1ifitconformswiththerule,and0ifitviolatestherule\.

Hereisanexampleofavalidanswer:

\[\["item\(0,red,block,upright,grounded\)","item\(1,blue,wedge,doorstop,touching\(0\)\)"\],0\]

\[\["item\(0,yellow,pyramid,upside\_down,grounded\)","item\(1,blue,wedge,doorstop,on\_top\_of\(0\)\)","item\(2,red,block,upright,pointing\(1\)\)"\],1\]

Returnyouranswerinthisexactformat:

\[\["item\(ID,color,shape,orientation,interaction\)",\.\.\.\],0\]

\[\["item\(ID,color,shape,orientation,interaction\)",\.\.\.\],1\]

Over the course of the game, the Agent revises the initial hypotheses using the following prompt:

Astructurehasoneormoreblocks\.Eachblockshouldcontainthefollowingattributes:

\-color:red,blue,yellow;

\-shape:block,wedge,pyramid;

\-orientation:upright,upside\_down,flat,cheesecake,doorstop;

\-relation:grounded\(whetherthepieceistouchingtheground\),touching\(ID\)\(whetherthepieceistouchinganotherpiecewithID\),pointing\(ID\)\(whetherthepieceispointingtoanotherpiecewithID\),on\_top\_of\(ID\)\(whetherthepieceisontopofanotherpiecewithID\)\.

Wedgescanbedoorstop,cheesecakeorflat,whilethetwoothershapescanbeflatbutnotcheesecakeordoorstop\.

Interactionscanbe:grounded,touching\(ID\),pointing\(ID\)andon\_top\_of\(ID\),whereIDisthefirstfieldofanotherpiece,e\.g\."pointing\(2\)"meansthispieceispointingtothepiecewithID2\.

Exampleofrulemodifications:

Quantifierchange:’Theremustbeablueblock’\-\>’Therearetwoblueblocks’

Additionalattribute:’Theremustbeablueblock’\-\>’Theremustbeablueblockthatisupright’

Attributechange:’Theremustbeablueblock’\-\>’Theremustbeayellowblock’

Thesemodificationsare"local":onlyoneattribute/quantifierischangedoraddedforeachmodification\.

Pleasemodifytherule’\{h\}’\.Generate\{num\}rulesforeachtypeofmodification\(Quantifierchange,Additionalattribute,Attributechange\)sothattheappendedimageofastructureis\{text\_y\}agoodstructure\(followstherule\):

Maketheformatanumberedlist\(1\.,2\.,\.\.\.,5\.\)Rememberthatthenewrulesshouldbea"local"modificationfromtherule’\{h\}’\.Donotuseattributevaluesthatarenotmentionedearlier\.Donotsayanythingotherthanthemodifiedrules\.