LEGO-Anything: Coding Agents for 3D Scene Reconstruction

Hugging Face Daily Papers Papers
3d-reconstruction coding-agents blender benchmark gpt-6-astra image-to-code

Summary

LEGO-Anything presents an image-to-code framework where coding agents iteratively write Blender code to reconstruct 3D scenes from single images, evaluated on the new LEGO-Bench benchmark, with LEGO-Plugin enhancing performance.

A 3D scene reconstructed from a single image is most useful when represented not as a rendering or a fixed 3D output, but as an explicit scene program whose execution yields a scene that can be inspected, edited, and queried. We present LEGO-Anything, an Image-to-Code framework in which a coding agent iteratively writes and executes Blender code, inspects scenes and renderings, and revises the program. To evaluate end-to-end scene recovery, we introduce LEGO-Bench, a simulator-grounded benchmark with 208 images from 104 diverse indoor and outdoor scenes. LEGO-Bench separately scores artifact validity, visible-surface geometry, and rendered appearance. Its simulator-grounded design enables extensibility and precise automatic evaluation. Among evaluated agents, GPT-6-astra achieves the strongest overall results, with 53.4% indoor and 39.6% outdoor scores, yet substantial gaps remain between delivering valid scene artifacts and faithfully recovering scene geometry and appearance. Analysis of agent construction trajectories reveals three recurring issues: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. These findings motivate LEGO-Plugin, a training-free harness plugin for more controlled iterative scene construction, which improves all six evaluated models, with relative gains of up to 62.7% in overall score. Finally, we test whether reconstructed scenes can represent natural images and support vision tasks. In LEGO-World, we derive object detections, instance masks, and relative depth as deterministic queries on scenes reconstructed by GPT-6-astra. These readouts show non-trivial performance across all three tasks but fall well short of specialized vision models, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.
Original Article
View Cached Full Text

Cached at: 09/30/26, 04:17 AM

Paper page - LEGO-Anything: Coding Agents for 3D Scene Reconstruction

Source: https://huggingface.co/papers/2609.36380

Abstract

A3Dscenereconstructedfromasingleimageismostusefulwhenrepresentednotasarenderingorafixed3Doutput,butasanexplicitsceneprogramwhoseexecutionyieldsascenethatcanbeinspected,edited,andqueried.WepresentLEGO-Anything,anImage-to-CodeframeworkinwhichacodingagentiterativelywritesandexecutesBlendercode,inspectsscenesandrenderings,andrevisestheprogram.Toevaluateend-to-endscenerecovery,weintroduceLEGO-Bench,asimulator-groundedbenchmarkwith208imagesfrom104diverseindoorandoutdoorscenes.LEGO-Benchseparatelyscoresartifactvalidity,visible-surfacegeometry,andrenderedappearance.Itssimulator-groundeddesignenablesextensibilityandpreciseautomaticevaluation.Amongevaluatedagents,GPT-6-astraachievesthestrongestoverallresults,with53.4%indoorand39.6%outdoorscores,yetsubstantialgapsremainbetweendeliveringvalidsceneartifactsandfaithfullyrecoveringscenegeometryandappearance.Analysisofagentconstructiontrajectoriesrevealsthreerecurringissues:weaksceneinitialization,regressiveeditsduringiteration,andunreliableself-evaluation.ThesefindingsmotivateLEGO-Plugin,atraining-freeharnesspluginformorecontrollediterativesceneconstruction,whichimprovesallsixevaluatedmodels,withrelativegainsofupto62.7%inoverallscore.Finally,wetestwhetherreconstructedscenescanrepresentnaturalimagesandsupportvisiontasks.InLEGO-World,wederiveobjectdetections,instancemasks,andrelativedepthasdeterministicqueriesonscenesreconstructedbyGPT-6-astra.Thesereadoutsshownon-trivialperformanceacrossallthreetasksbutfallwellshortofspecializedvisionmodels,suggestingthatprogram-constructedscenesfromcurrentcodingagentsareapromisingbutnotyetsufficientlypreciserepresentationofnaturalimages.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.36380

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.36380 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.36380 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.36380 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

Hugging Face Daily Papers

4DCodeBench introduces a benchmark for 4D inverse graphics through code generation, where agents reconstruct dynamic scenes from video as executable graphics programs. Frontier models show strong static reconstruction but struggle to reliably capture complex dynamics like deformation, fluid flow, and fracture.

3DCodeVerse: Exploring 3D Worlds as Code

Hugging Face Daily Papers

3DCodeVerse introduces a research ecosystem with a 2M-scale dataset of executable 3D code, a multi-agent DAG harness for automated generation and verification of 3D content, and empirical guidance showing fine-tuned open models achieve a 90% win rate against Claude Design (3D Object).

BrickBench: Evaluating Agentic Brick Design

Hugging Face Daily Papers

We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design, where an agent must produce a physically buildable assembly from a discrete parts library while satisfying semantic and design criteria. The paper also introduces BrickAgent, an environment for coding agents, and finds leading agents meet physical requirements but still fall short of human designs.

Show HN: Made an open-source Lego AI generator

Hacker News Top

Show HN: an open-source project (ldraw-nova) that lets an AI agent design buildable LEGO models by writing LDraw code, exporting 3D views, VR-ready glTF files, and agent thinking traces, with Docker-based web app setup.