2.5-D Decomposition for LLM-Based Spatial Construction

# 2.5-D Decomposition for LLM-Based Spatial Construction
Source: [https://arxiv.org/html/2605.07066](https://arxiv.org/html/2605.07066)
###### Abstract

Autonomous systems that build structures from natural\-language instructions need reliable spatial reasoning, yet large language models \(LLMs\) make systematic coordinate errors when generating three\-dimensional block placements\. We present a neuro\-symbolic pipeline based on*2\.5\-D decomposition*: the LLM plans in the two\-dimensional horizontal plane while a deterministic executor computes all vertical placement from column occupancy, eliminating an entire class of errors\. On the Build What I Mean benchmark \(160 rounds\), GPT\-4o\-mini with this pipeline achieves 94\.6% mean structural accuracy across 12 independent runs, within 3\.0 percentage points of the 97\.6% ceiling imposed by architect\-agent errors that no builder\-side improvement can address\. This outperforms both GPT\-4o at 90\.3% and the best competing system at 76\.3%\. A controlled ablation confirms that 2\.5\-D decomposition is the dominant contributor, accounting for 50\.7 percentage points of accuracy\. The pipeline transfers directly to edge hardware: Nemotron\-3 120B running locally on an NVIDIA Jetson Thor AGX matches the cloud result at 94\.5% with no prompt modifications\. The underlying principle, removing deterministic dimensions from the LLM’s output space, applies to any autonomous construction or assembly task where gravity or other physical constraints fix one or more degrees of freedom\. A transfer experiment on 500 IGLU collaborative building tasks confirms the effect generalizes beyond the primary benchmark\.

![Refer to caption](https://arxiv.org/html/2605.07066v1/x1.png)Figure 1:A benchmark round requiring T\-shape recognition\. Instruction: “Keeping the T shape, extend the existing green structure by adding two green blocks to the longer base\. Then add one purple block to each arm\.” New blocks are marked with\+\+\.## IIntroduction

Autonomous systems that construct physical structures from high\-level instructions must solve two problems: understanding what to build and computing where to place each component\. Large language models \(LLMs\) are effective at the first problem but unreliable at the second\. When asked to produce three\-dimensional coordinates, LLMs make systematic errors in vertical placement, off\-by\-one stacking, and duplicate positions\[[2](https://arxiv.org/html/2605.07066#bib.bib2),[3](https://arxiv.org/html/2605.07066#bib.bib3)\], consistent with LeCun’s observation that LLMs lack internal world models for enforcing physical constraints\[[20](https://arxiv.org/html/2605.07066#bib.bib20)\]\.

We observe that many construction domains exhibit a 2\.5\-D structure: one or more output dimensions are not free variables but deterministic functions of the others and the current state\. In gravity\-constrained block construction, the vertical coordinate of any new block is fully determined by the column occupancy below it\. The LLM does not need to reason about this axis and in practice cannot do so reliably\.

Our approach separates the problem accordingly\. The LLM produces a plan in the two\-dimensional horizontal plane, specifying only\(x,z\)\(x,z\)positions, colors, and action types\. A deterministic spatial executor computes all vertical placement\. This 2\.5\-D decomposition eliminates an entire class of coordinate errors\. Fig\.[1](https://arxiv.org/html/2605.07066#S0.F1)shows a representative round in which the agent must recognize a T\-shaped structure and extend it while preserving symmetry\.

The system includes four additional components: a structure analyzer that detects geometric primitives in the existing grid, an underspecification detector that decides when to ask a clarification question based on expected\-value analysis, a peephole prompt optimizer that injects pattern\-specific corrections before the LLM call, and a rule\-based plan verifier\.

This principle extends beyond block construction to any autonomous system where physical constraints make some output dimensions computable from others\. In robotic construction and assembly, gravity determines stacking order\. In terrain\-following UAV path planning, altitude is a function of horizontal position and terrain data\. In automated equipment maintenance, physical constraints on part placement reduce the effective degrees of freedom\. The common pattern is that removing deterministic dimensions from the language model’s output space improves reliability without reducing the system’s capability\.

An important ceiling on this benchmark is the architect agent that answers clarification questions: it is itself an LLM and sometimes provides incorrect information\. Error analysis shows that 54\.8% of residual failures originate in the architect, bounding the achievable accuracy at approximately 97\.6% regardless of builder quality\. Our 94\.6% result leaves only 3\.0 percentage points of real headroom\.

Our contributions are: \(1\) a 2\.5\-D decomposition that restricts the LLM to horizontal planning while a deterministic executor handles vertical placement, \(2\) a peephole prompt optimization methodology that targets systematic LLM failure modes with pattern\-matched corrections, \(3\) an empirical demonstration that this approach enables GPT\-4o\-mini to outperform GPT\-4o on block construction accuracy \(p<0\.0001p<0\.0001\), and \(4\) a decision\-theoretic framework for underspecification handling under an asymmetric scoring function\.

## IIRelated Work

LLM spatial reasoning\.Yamada et al\.\[[2](https://arxiv.org/html/2605.07066#bib.bib2)\]benchmark LLMs on spatial tasks and find that chain\-of\-thought prompting improves accuracy\. Bang et al\.\[[3](https://arxiv.org/html/2605.07066#bib.bib3)\]document persistent failures in relative positioning for GPT\-family models\. Our work differs in that we do not attempt to improve the LLM’s spatial reasoning directly\. Instead, we restrict the LLM to a lower\-dimensional output space and handle the eliminated dimension with deterministic code\.

Plan\-then\-execute and neuro\-symbolic decomposition\.Wang et al\.\[[4](https://arxiv.org/html/2605.07066#bib.bib4)\]show that separating planning from execution improves zero\-shot reasoning, and Khot et al\.\[[5](https://arxiv.org/html/2605.07066#bib.bib5)\]split complex tasks into sub\-problems handled by specialized modules\. Yi et al\.\[[6](https://arxiv.org/html/2605.07066#bib.bib6)\]decompose visual question answering into a neural perception module and a symbolic execution engine\. Our pipeline adopts these ideas but extends them with a dimensional constraint: the planning module operates in 2D while execution operates in 3D, a separation not explored in prior decomposed\-prompting work\.

LLM\-guided robotic execution\.Recent work constrains LLM outputs to what deterministic or learned modules can execute\. Ahn et al\.\[[14](https://arxiv.org/html/2605.07066#bib.bib14)\]ground language model plans in robotic affordances, Liang et al\.\[[15](https://arxiv.org/html/2605.07066#bib.bib15)\]have the LLM generate policy code that calls perception APIs directly, and Huang et al\.\[[16](https://arxiv.org/html/2605.07066#bib.bib16)\]close the loop with environment feedback\. Our approach shares this principle but exploits a dimensional constraint specific to gravity\-bound construction: we remove an entire spatial axis from the LLM’s output space and compute it deterministically\.

3D construction agents\.Wang et al\.\[[11](https://arxiv.org/html/2605.07066#bib.bib11)\]present VOYAGER, an LLM\-powered Minecraft agent for open\-ended exploration\. Zhu et al\.\[[12](https://arxiv.org/html/2605.07066#bib.bib12)\]propose GITM for generally capable open\-world agents\. These systems target sequential decision\-making rather than precise structural construction from underspecified instructions\. The BWIM task requires coordinate\-level accuracy against a target configuration, making coordinate arithmetic errors the dominant failure mode\.

## IIIProblem Formulation

### III\-ATask Definition

The BWIM benchmark\[[1](https://arxiv.org/html/2605.07066#bib.bib1)\]defines a block construction task on a discrete grid𝒢=\{0,…,8\}×\{0,…,4\}×\{0,…,8\}\\mathcal\{G\}=\\\{0,\\ldots,8\\\}\\times\\\{0,\\ldots,4\\\}\\times\\\{0,\\ldots,8\\\}, where the axes correspond to width \(xx\), height \(yy, vertical\), and depth \(zz\)\. Each cell is either empty or occupied by a block of colorc∈\{red,blue,green,orange,yellow,purple\}c\\in\\\{\\text\{red\},\\text\{blue\},\\text\{green\},\\text\{orange\},\\text\{yellow\},\\text\{purple\}\\\}\.

In each round, the builder agent receives a natural\-language instructionIIand a starting grid stateG0G\_\{0\}, and must produce a target grid stateG∗=G0∪BG^\{\*\}=G\_\{0\}\\cup BwhereBBis the set of new block placements\. The agent may issue one clarification question before building\. The scoring function awards\+10\+10for a correct build,−10\-10for an incorrect build, and−5\-5for each question asked \(see Fig\.[1](https://arxiv.org/html/2605.07066#S0.F1)for an example round\)\.

### III\-B2\.5\-D Decomposition

The construction grid is a 2\.5\-D domain in the sense of Marr\[[8](https://arxiv.org/html/2605.07066#bib.bib8)\]: the vertical axis is a deterministic function of horizontal position and existing occupancy, analogous to 2\.5\-D machining where the tool path is free in two axes but the third changes only in discrete, computed steps\[[9](https://arxiv.org/html/2605.07066#bib.bib9)\]\. The grid enforces a gravity constraint: a block at\(x,y,z\)\(x,y,z\)can only exist ify=0y=0or a block exists at\(x,y−1,z\)\(x,y\-1,z\)\. The vertical coordinate of any new block is:

y∗​\(x,z,G\)=min⁡\{y∈\{0,…,4\}∣\(x,y,z\)∉dom​\(G\)\}y^\{\*\}\(x,z,G\)=\\min\\\{y\\in\\\{0,\\ldots,4\\\}\\mid\(x,y,z\)\\notin\\text\{dom\}\(G\)\\\}\(1\)This reduces the LLM’s output space from\|𝒢\|×\|𝒞\|\|\\mathcal\{G\}\|\\times\|\\mathcal\{C\}\|to\|\{0,…,8\}\|2×\|𝒞\|\|\\\{0,\\ldots,8\\\}\|^\{2\}\\times\|\\mathcal\{C\}\|, eliminatingyy\-coordinate errors entirely\.

## IVArchitecture

The agent processes each instruction through a six\-stage pipeline:parse,analyze,plan,verify,execute, andformat\. If any stage raises an unrecoverable error, control falls back to a direct LLM call with an engineered system prompt\.

1. 1\.Instruction Parser\.Extracts the building directive from the incoming message and normalizes the representation\.
2. 2\.Structure Analyzer\.Detects geometric primitives \(rows, stacks, L\-shapes, T\-shapes\) inG0G\_\{0\}and produces a structured description injected into the planner prompt\.
3. 3\.Build Planner \(LLM\)\.Decomposes the instruction into a planP=⟨a1,…,ak⟩P=\\langle a\_\{1\},\\ldots,a\_\{k\}\\rangleof typed JSON actions, each specifying an action type, horizontal position\(xi,zi\)\(x\_\{i\},z\_\{i\}\), colorcic\_\{i\}, and countnin\_\{i\}\. Nine worked examples in the system prompt\[[7](https://arxiv.org/html/2605.07066#bib.bib7)\]cover chains, L\-shapes, T\-shapes, and edge placements\. No example includesyy\-coordinates\.
4. 4\.Plan Verifier\.A rule\-based module validates the plan against the instruction text andG0G\_\{0\}with four correction passes: direction consistency, endpoint cap correction, T\-shape extend correction, and stacking plausibility\.
5. 5\.Spatial Executor\.A deterministic engine processes each action on an in\-memory grid, resolving relative references, computingyyvia \([1](https://arxiv.org/html/2605.07066#S3.E1)\), and chaining positional context\. No LLM call is involved\. A same\-color skip\-forward rule in the extend handler detects occupied start positions and advances one grid step, preventing vertical stacking when horizontal extension is intended\.
6. 6\.Response Formatter\.Translates the final grid state into the protocol\-required output format\.

![Refer to caption](https://arxiv.org/html/2605.07066v1/x2.png)Figure 2:2\.5\-D decomposition: the LLM planner generates 2D plans with\(x,z\)\(x,z\)coordinates and action types\. The deterministic executor computes vertical placement via column occupancy \(Equation[1](https://arxiv.org/html/2605.07066#S3.E1)\)\. Sequence order encodes vertical \(yy\) position\.Fig\.[2](https://arxiv.org/html/2605.07066#S4.F2)illustrates the decomposition\. A plan specifying “stack 3 red at\(5,6\)\(5,6\), place red at\(6,6\)\(6,6\)” contains noyy\-coordinates\. The executor computes a three\-block column atx=5x=5\(withy=0,1,2y=0,1,2\) and a single block atx=6x=6\(aty=0y=0\)\.

In our initial approach, the LLM produced the complete 3D output directly, which required it to enumerate every block’s\(x,y,z\)\(x,y,z\)coordinates and chain height calculations across stacking operations\. That approach suffered from frequent off\-by\-oneyy\-errors, duplicate placements, and miscounted heights\. As shown in Section[VI](https://arxiv.org/html/2605.07066#S6), the current approach eliminates these failure modes entirely\.

### IV\-AUnderspecification Handling

Many BWIM instructions deliberately omit color or block count\. The scoring function creates an asymmetric decision problem\. Letppdenote the probability of a correct guess:

EVguess​\(p\)\\displaystyle\\text\{EV\}\_\{\\text\{guess\}\}\(p\)=20​p−10\\displaystyle=20p\-10\(2\)EVask\\displaystyle\\text\{EV\}\_\{\\text\{ask\}\}=20​pa−15\\displaystyle=20p\_\{a\}\-15\(3\)Setting these equal withpa≈1p\_\{a\}\\approx 1yields the indifference thresholdp∗=3/4=0\.75p^\{\*\}=3/4=0\.75\. Below this threshold, asking dominates guessing\.

Missing color\.Heuristic analysis determines whether a color can be inferred from context \(for example, reusing the sole color mentioned\) or is genuinely ambiguous\. When ambiguous, the agent issues a clarification question\.

Missing count\.Heuristic count inference \(copying an adjacent stack’s height or defaulting to three\) achieves approximately 65% accuracy on BWIM instructions, well belowp∗p^\{\*\}, so the agent asks whenever a count is underspecified and a question has not already been used in the current round\.

A complication is that the architect answering questions is itself an LLM that receives only the target structure as coordinates, not the original instruction\. Generic questions such as “How many blocks should be in the stack?” produce a 23% error rate because the architect cannot identify which stack is referenced\. The agent generates color\-specific questions that name the block color, allowing the architect to identify the correct stack by counting blocks of that color in its coordinate data\.

Because only one question is permitted per round, the agent prioritizes color over count \(color errors are harder to recover from heuristically\)\. When multiple phrases lack counts, the remainder fall back to a three\-level heuristic cascade: copy the height of an adjacent stack, use the tallest stack on the grid, or default to three \(the modal count in the benchmark, observed in 53% of count\-underspecified trials\)\.

### IV\-BPeephole Prompt Optimization

Certain spatial concepts \(“each end,” chain references, L\-shape extensions\) are disproportionately error\-prone for GPT\-4o\-mini\. These errors are systematic: specific input patterns reliably trigger specific failure modes\. Borrowing from compiler design, where a peephole optimizer matches known suboptimal instruction sequences and replaces them with better ones\[[10](https://arxiv.org/html/2605.07066#bib.bib10)\], we scan each instruction against 15 pattern\-matching rules\. When a rule fires, a targeted correction with a worked example is injected into the prompt before the LLM call\. Rules are independent and composable\.

This methodology is domain\-transferable: identify recurring failure modes, classify by input trigger, write micro\-corrections with worked examples, inject dynamically via pattern matching, and validate against regressions\. It applies to any domain where LLMs make systematic, input\-predictable errors\.

Table[I](https://arxiv.org/html/2605.07066#S4.T1)lists five representative rules from the set of 15\. Each rule matches a trigger phrase, identifies the systematic error the LLM produces for that phrase, and injects a targeted correction with a coordinate example\.

TABLE I:Representative peephole prompt optimization rules\.To guard against overfitting, all few\-shot examples use different colors, counts, positions, and phrasing from the benchmark instructions\. We measure unigram and bigram cosine similarity between each enrichment example and every benchmark instruction\. The maximum similarity between any example and any benchmark instruction is 0\.71, while benchmark instructions reach pairwise similarities of up to 0\.80 among themselves\. The enrichment examples share less surface overlap with any evaluation item than the evaluation items share with each other\.

## VExperimental Setup

We evaluate on the BWIM benchmark’s 160\-round scenario, which pairs the builder agent with both a rational architect \(which answers questions correctly\) and an unreliable architect \(which sometimes provides incorrect information\)\. All runs use the same codebase and differ only in model selection or component ablation\.

Models\.The primary model is GPT\-4o\-mini \(2024\-07\-18\)\. We include GPT\-4o \(2024\-08\-06\) as a comparison to assess whether model scale provides additional gains beyond the deterministic pipeline\. All prompt engineering was developed against GPT\-4o\-mini\. The GPT\-4o runs swap only the model identifier; all prompts, peephole rules, temperature settings, and pipeline logic are unchanged\. We also evaluate NVIDIA Nemotron\-3\-Super\-120B\-A12B\[[17](https://arxiv.org/html/2605.07066#bib.bib17)\], an open\-weight model quantized to NVFP4, served locally via vLLM\[[19](https://arxiv.org/html/2605.07066#bib.bib19)\]on an NVIDIA Jetson Thor AGX Developer Kit\[[18](https://arxiv.org/html/2605.07066#bib.bib18)\]\(14\-core Arm Neoverse V3AE CPU, 2560\-core Blackwell GPU with fifth\-generation Tensor Cores\)\. No model\-specific prompt tuning was performed for the Nemotron runs\.

Metrics\.We report structural accuracy, composite score \(accuracy\-weighted with question penalties\), and block\-level F1\. Each round’s predicted and target placements are compared as sets of \(color,xx,yy,zz\) tuples\. F1 is the harmonic mean of precision and recall over these sets, equivalent to the Dice coefficient\[[21](https://arxiv.org/html/2605.07066#bib.bib21)\], providing graded partial credit for near\-miss rounds that binary accuracy marks as failures\. For GPT\-4o\-mini, we report statistics across 12 independent runs\. For GPT\-4o and Nemotron\-3, across six runs\.

Frozen codebase\.All reported runs use the same frozen code commit\. No code changes were made between runs within each model condition\. This ensures that variance reflects only LLM non\-determinism, not development changes\. The pipeline calls the LLM at most twice per round \(planning plus at most one clarification re\-plan\)\.

## VIResults

### VI\-AComparison with Baselines

TABLE II:Comparison with baselines and competing systems\. hisandan/build\-it\-3 prompts GPT\-4o directly for coordinate output without a decomposition pipeline; result is from a single public leaderboard submission \(n=1n=1\), score computed from the public run log\.Table[II](https://arxiv.org/html/2605.07066#S6.T2)shows our system compared with baselines and the top competing system on the BWIM leaderboard\. The prompt\-engineered row reflects 16 runs with iterative prompt tuning and few\-shot examples but no 2\.5\-D decomposition \(n=16n=16, mean 63\.3%,σ=8\.0%\\sigma=8\.0\\%\)\. This represents the practical ceiling of prompt engineering alone with GPT\-4o\-mini\. Our system outperforms the competing system, which uses GPT\-4o \(a larger and more expensive model\), by 18\.3 percentage points\.

### VI\-BAblation Study

TABLE III:Ablation study \(n=7n=7per condition, GPT\-4o\-mini\)\.Table[III](https://arxiv.org/html/2605.07066#S6.T3)presents an ablation of the final system\. Removing 2\.5\-D decomposition produces the largest accuracy drop \(−50\.7\-50\.7pp,p<0\.0001p<0\.0001\), reducing accuracy to 43\.8%, comparable to the bare LLM baseline\. Underspecification questions contribute−17\.9\-17\.9pp, peephole prompt optimization−11\.3\-11\.3pp, and the skip\-forward rule−3\.9\-3\.9pp, all significant atp<0\.0001p<0\.0001\. The plan verifier shows no significant effect \(−0\.6\-0\.6pp,p=0\.36p=0\.36\)\. Even with enrichment entirely disabled, the system reaches 83\.3%, still 7 pp above the best competing system’s 76\.3% with GPT\-4o\.

### VI\-CModel Comparison

TABLE IV:Model comparison on the frozen pipeline\.Table[IV](https://arxiv.org/html/2605.07066#S6.T4)compares GPT\-4o\-mini and GPT\-4o on the identical pipeline\. GPT\-4o\-mini outperforms GPT\-4o by 4\.3 pp with non\-overlapping 95% confidence intervals \(Welchtt\-test:t=14\.3t=14\.3,𝑑𝑓=13\.5\\mathit\{df\}=13\.5,p<0\.0001p<0\.0001, Cohen’sd=6\.8d=6\.8\)\. Block\-level F1 exceeds 0\.97 for all models, indicating that even incorrect rounds are typically near\-misses \(a single misplaced block\) rather than catastrophic failures\. All prompt engineering was developed against GPT\-4o\-mini due to its 16×\\timeslower token cost\. Each 160\-round GPT\-4o\-mini run completes in approximately 8\.9 minutes at an estimated cost of $0\.07\. GPT\-4o runs take approximately 10\.1 minutes at $1\.10 per run, roughly 16×\\timesmore expensive for lower accuracy on this pipeline\. The cost difference made GPT\-4o impractical as the development model for iterative prompt engineering across hundreds of test rounds\. GPT\-4o with the untuned pipeline still reaches 90\.3%, well above the best competing system’s 76\.3% using GPT\-4o without a decomposition pipeline, indicating that the pipeline architecture provides value independent of model\-specific tuning\.

### VI\-DEdge Inference Latency

Across six runs, Nemotron\-3\-Super\-120B\-A12B on a local NVIDIA Jetson Thor AGX achieved 94\.5% mean accuracy \(σ=0\.83%\\sigma=0\.83\\%, \+943 mean score\), statistically indistinguishable from the GPT\-4o\-mini cloud mean \(Welcht=0\.26t=0\.26,p=0\.80p=0\.80\)\. The model ran at NVFP4 precision with no prompt modifications\. Median per\-request latency was 50\.2s \(P95: 130\.6s,n=158n=158planner calls\), totaling 162 minutes for the 160\-round run compared to 8\.9 minutes via cloud API, an 18×\\timesslowdown acceptable for non\-real\-time construction tasks\. The result demonstrates that the 2\.5\-D decomposition pipeline is not tightly coupled to a specific LLM provider and can transfer to locally hosted open\-weight models on embedded AI hardware with minimal accuracy loss\.

### VI\-ETransfer to IGLU

To test generality, we applied the 2\.5\-D decomposition to the IGLU collaborative building dataset\[[22](https://arxiv.org/html/2605.07066#bib.bib22)\], an independent benchmark with different instructions, a larger grid \(11×9×1111\\times 9\\times 11\), and no shared evaluation items with BWIM\. We evaluated 500 gravity\-compatible tasks with GPT\-4o\-mini at temperature 0, comparing a bare coordinate\-output prompt against the 2\.5\-D decomposed prompt\. The decomposition improved mean block\-level F1 from 0\.723 to 0\.798 \(pairedt​\(499\)=5\.76t\(499\)=5\.76,p<10−8p<10^\{\-8\},d=0\.26d=0\.26, 95% CI\[0\.050,0\.101\]\[0\.050,0\.101\]\)\. No enrichment rules, plan verifier, or underspecification handling were used\. The result confirms that dimensional reduction alone, the core architectural principle, transfers to a structurally different spatial construction task\.

### VI\-FError Analysis

Across all 12 GPT\-4o\-mini runs \(1,920 total rounds\), 104 rounds produce incorrect structures \(5\.4% failure rate\)\. Of these, 57 \(54\.8%\) originate in the architect agent providing incorrect color information in response to clarification questions\. The remaining 47 \(45\.2%\) are builder errors: 25 spatial reasoning mistakes and 22 incorrect color\-position assignments\. Excluding architect errors, the builder pipeline achieves 97\.6% accuracy \(1,873/1,920 rounds correct\)\.

## VIILimitations

The primary evaluation uses a single 160\-round benchmark \(BWIM\), though Section V\-F confirms transfer to an independent dataset\. The 2\.5\-D decomposition depends on a gravity constraint that makes vertical coordinates computable\. Tasks without such a constraint would require a different decomposition\. The 15 peephole enrichment rules were developed by analyzing failure modes on the BWIM instruction set\. While the targeted failure patterns are general spatial reasoning problems, their coverage of other instruction distributions is unknown\. The competing system’s lower accuracy with GPT\-4o \(76\.3%\) compared to our GPT\-4o\-mini result \(94\.6%\) suggests genuine structural benefit beyond benchmark\-specific tuning\. The error analysis shows that 54\.8% of failures originate in the architect agent, creating a 3\.0 pp accuracy ceiling that builder\-side improvements alone cannot close\.

## VIIIConclusion and Future Work

We presented a neuro\-symbolic approach to LLM\-based block construction in which the LLM plans in a two\-dimensional horizontal plane while a deterministic executor computes all vertical placement\. On the BWIM benchmark, this 2\.5\-D decomposition enables GPT\-4o\-mini to achieve 94\.6% mean structural accuracy, outperforming GPT\-4o at 90\.3% \(p<0\.0001p<0\.0001\)\. A controlled ablation study confirms that 2\.5\-D decomposition accounts for the largest single contribution \(−50\.7\-50\.7pp when removed\)\.

The broader principle is that LLM spatial reasoning can be made reliable by identifying which dimensions of the output space are deterministically computable from task constraints and removing those dimensions from the LLM’s responsibility\. This principle applies to any domain where one or more output dimensions are deterministic functions of the others, a condition we term 2\.5\-D structure by analogy with Marr’s 2\.5\-D sketch\[[8](https://arxiv.org/html/2605.07066#bib.bib8)\]\.

For autonomous systems, this decomposition strategy is relevant wherever physical laws constrain assembly or placement\. Gravity\-constrained robotic construction, terrain\-following path planning, and automated equipment assembly all share the property that one or more spatial dimensions are determined by physics rather than by choice\. Consider a concrete example: in terrain\-following UAV mission planning, the vehicle must maintain a fixed altitude above ground level\. The horizontal flight path\(x,z\)\(x,z\)is a free planning variable, but the altitudeyyis a deterministic function of position and the digital elevation model,y∗​\(x,z\)=DEM​\(x,z\)\+hclearancey^\{\*\}\(x,z\)=\\text\{DEM\}\(x,z\)\+h\_\{\\text\{clearance\}\}\. An LLM\-based mission planner could specify waypoints in the horizontal plane while a deterministic module computes the altitude profile, exactly paralleling our 2\.5\-D decomposition\. The same pattern applies to automated maintenance on fixed equipment, where part insertion paths are constrained by the machine geometry and only the selection and sequencing of parts is a free variable\.

Identifying and exploiting these deterministic dimensions offers a path toward reliable LLM\-guided autonomous operation in physically constrained environments\.

Future work includes developing methods for the builder to detect architect errors and investigating whether the 2\.5\-D principle extends to tasks with structural stability or connectivity constraints\.

## Reproducibility

Source code, prompts, enrichment rules, evaluation scripts, and scoring logs are available at[https://github\.com/paulwhitten/AgentWhetters\-bwim](https://github.com/paulwhitten/AgentWhetters-bwim)\(tagv1\.0\.5\)\. The IGLU transfer experiment code is iniglu/on themainbranch of the same repository\. Models:gpt\-4o\-mini\-2024\-07\-18,gpt\-4o\-2024\-08\-06\(OpenAI API\), andNemotron\-3\-Super\-120B\-A12B\-NVFP4\(local, vLLM\[[19](https://arxiv.org/html/2605.07066#bib.bib19)\], Jetson Thor AGX Developer Kit, Blackwell GPU, Arm Neoverse V3AE CPU\)\. Build planner temperature 0\.1; all other calls temperature 0\.2\. The benchmark’s architect \(green\) agent usesgpt\-4o\-mini\.

## References

- \[1\]UvA LTL, “Build what I mean,” 2026\. \[Online\]\. Available:[https://github\.com/ltl\-uva/build\_what\_i\_mean](https://github.com/ltl-uva/build_what_i_mean)
- \[2\]Y\. Yamada, Y\. Bao, A\. K\. Lampinen, J\. Kasai, and I\. Yildirim, “Evaluating spatial understanding of large language models,”Trans\. Mach\. Learn\. Res\., 2024\.
- \[3\]Y\. Banget al\., “A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,” inProc\. AACL, 2023\.
- \[4\]L\. Wanget al\., “Plan\-and\-solve prompting: Improving zero\-shot chain\-of\-thought reasoning by large language models,” inProc\. ACL, 2023\.
- \[5\]T\. Khotet al\., “Decomposed prompting: A modular approach for solving complex tasks,” inProc\. ICLR, 2023\.
- \[6\]K\. Yiet al\., “Neural\-symbolic VQA: Disentangling reasoning from vision and language understanding,” inProc\. NeurIPS, 2018\.
- \[7\]J\. Weiet al\., “Chain\-of\-thought prompting elicits reasoning in large language models,” inProc\. NeurIPS, 2022\.
- \[8\]D\. Marr,Vision: A Computational Investigation into the Human Representation and Processing of Visual Information\. W\.H\. Freeman, 1982\.
- \[9\]A\. Nayak, J\. Steuben, D\. Poff, M\. Kirby, and H\. Ilies, “Automatic 2\.5D part decomposition for multi\-axis machining,”Comput\.\-Aided Des\., 2015\.
- \[10\]W\. M\. McKeeman, “Peephole optimization,”Commun\. ACM, vol\. 8, no\. 7, pp\. 443–444, 1965\.
- \[11\]G\. Wanget al\., “VOYAGER: An open\-ended embodied agent with large language models,”arXiv:2305\.16291, 2023\.
- \[12\]X\. Zhuet al\., “Ghost in the Minecraft: Generally capable agents for open\-world environments via large language models with text\-based knowledge and memory,”arXiv:2305\.17144, 2023\.
- \[13\]hisandan, “build\-it\-3: BWIM competition agent,” 2026\. \[Online\]\. Available:[https://github\.com/hisandan/build\-it\-3](https://github.com/hisandan/build-it-3)\(accessed Apr\. 2026\)
- \[14\]M\. Ahnet al\., “Do as I can, not as I say: Grounding language in robotic affordances,” inProc\. CoRL, 2022\.
- \[15\]J\. Lianget al\., “Code as policies: Language model programs for embodied control,” inProc\. IEEE ICRA, 2023\.
- \[16\]W\. Huanget al\., “Inner monologue: Embodied reasoning through planning with language models,”arXiv:2207\.05608, 2022\.
- \[17\]NVIDIA, “Nemotron\-3\-Super\-120B\-A12B,” 2024\. \[Online\]\. Available:[https://huggingface\.co/nvidia/Nemotron\-3\-Super\-120B\-A12B\-NVFP4](https://huggingface.co/nvidia/Nemotron-3-Super-120B-A12B-NVFP4)
- \[18\]NVIDIA, “Jetson Thor,” 2025\. \[Online\]\. Available:[https://www\.nvidia\.com/en\-us/autonomous\-machines/embedded\-systems/jetson\-thor/](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/)
- \[19\]W\. Kwonet al\., “Efficient memory management for large language model serving with PagedAttention,” inProc\. SOSP, 2023\.
- \[20\]Y\. LeCun, “A path towards autonomous machine intelligence,” version 0\.9\.2, Tech\. Rep\., Meta AI, Jun\. 2022\. \[Online\]\. Available: https://openreview\.net/forum?id=BZ5a1r\-kVsf
- \[21\]L\. R\. Dice, “Measures of the amount of ecologic association between species,”Ecology, vol\. 26, no\. 3, pp\. 297–302, 1945\.
- \[22\]J\. Kiselevaet al\., “IGLU: Interactive grounded language understanding in a collaborative environment,” inProc\. NeurIPS Datasets and Benchmarks, 2022\.