BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization

# BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization
Source: [https://arxiv.org/html/2605.26182](https://arxiv.org/html/2605.26182)
Zhengyang Ni1,2,3,\*Feng Yan1,2,3,\*Yu Guo1,2,3Fei Wang1,2,3,† 1Xi’an Jiaotong University 2State Key Laboratory of Human\-Machine Hybrid Augmented Intelligence 3Institute of Artificial Intelligence and Robotics

###### Abstract

Generating physically buildable brick structures from 3D shapes requires more than geometric reconstruction: the output must also satisfy discrete part constraints and structural stability\. Existing brick generation methods either rely on heuristic optimization, which can break down when the target 3D shape does not admit a feasible structure under predefined constraints, or generate brick sequences without explicitly modeling the underlying 3D geometry and assembly relations\. In this work, we presentBrickAnything, a geometry\-conditioned autoregressive framework for generating buildable brick structures from diverse 3D representations\. BrickAnything uses point clouds as a unified geometric interface and predicts brick sequences that reconstruct the target shape under assembly constraints\. To model structural dependencies among bricks, we introduce a structure\-aware tree tokenization, which represents brick structures through local attachment relations\. This formulation makes sequence generation more consistent with the physical construction process, and reduces invalid intermediate states\. We further introduce preference\-based alignment post\-training, validity\-constrained decoding and adaptive rollback to improve buildability objectives such as stability and geometric fidelity\. Extensive experiments demonstrate that BrickAnything produces geometrically faithful and physically realizable brick structures, and that the proposed tokenization effectively reduces rollback and regeneration compared with conventional ordering strategies\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\.## 1Introduction

Discrete brick\-based structures offer a physically grounded and modular representation for constructing 3D objects, bridging digital geometry and real\-world assembly\. Unlike conventional 3D representations such as mesh\[[52](https://arxiv.org/html/2605.26182#bib.bib1),[46](https://arxiv.org/html/2605.26182#bib.bib4),[5](https://arxiv.org/html/2605.26182#bib.bib2),[49](https://arxiv.org/html/2605.26182#bib.bib3),[39](https://arxiv.org/html/2605.26182#bib.bib6),[23](https://arxiv.org/html/2605.26182#bib.bib5)\], point cloud\[[42](https://arxiv.org/html/2605.26182#bib.bib7),[15](https://arxiv.org/html/2605.26182#bib.bib11),[16](https://arxiv.org/html/2605.26182#bib.bib12),[35](https://arxiv.org/html/2605.26182#bib.bib8),[37](https://arxiv.org/html/2605.26182#bib.bib9),[51](https://arxiv.org/html/2605.26182#bib.bib10)\], Gaussian Splatting\[[18](https://arxiv.org/html/2605.26182#bib.bib13),[12](https://arxiv.org/html/2605.26182#bib.bib14),[3](https://arxiv.org/html/2605.26182#bib.bib16),[17](https://arxiv.org/html/2605.26182#bib.bib15)\], or implicit fields\[[27](https://arxiv.org/html/2605.26182#bib.bib18),[28](https://arxiv.org/html/2605.26182#bib.bib19),[47](https://arxiv.org/html/2605.26182#bib.bib20),[8](https://arxiv.org/html/2605.26182#bib.bib21)\], brick assemblies are governed by strict combinatorial and structural constraints\. A generated brick structure must not only approximate the target geometry, but also satisfy valid part compatibility, inter\-brick connectivity, and structural stability\. This combination of geometric fidelity and physical buildability makes brick generation a fundamentally different and more challenging problem than conventional 3D content synthesis\.

Existing methods for brick structure generation can be broadly categorized into generative approaches and heuristic or search\-based methods\. Recent generative models, such as BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\], LEGO®\-Maker\[[13](https://arxiv.org/html/2605.26182#bib.bib23)\]and BrickNet\[[19](https://arxiv.org/html/2605.26182#bib.bib24)\], formulate brick construction as a sequential prediction problem, enabling flexible and expressive generation of complex assemblies\. However, these methods typically rely on high\-level inputs such as text or images, which do not provide explicit 3D geometric constraints\. As a result, the generated structures are primarily guided by semantic priors or 2D visual cues, making it difficult to ensure accurate spatial configurations and faithful reconstruction of target shapes, especially for geometrically complex objects\. On the other hand, heuristic or search\-based methods, including Image2Lego\[[21](https://arxiv.org/html/2605.26182#bib.bib25)\]and Legolization\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\], explicitly incorporate geometric and structural constraints by formulating brick assembly as a combinatorial optimization problem\. While these methods can produce physically feasible structures, they often rely on fixed target representations and heuristic or search\-based optimization strategies\. This limits their flexibility when the target geometry is not directly realizable in the discrete brick space, and leads to high computational complexity as the search space grows\. This raises a natural question:Can we design a unified framework for brick generation that explicitly leverages 3D geometry, adapts to diverse input representations, and generates buildable structures in a more flexible and effective manner?

To address this question, we proposeBrickAnything, a unified framework for geometry\-conditioned buildable brick generation\. We adopt point clouds as a modality\-agnostic intermediate representation, allowing diverse 3D inputs to be mapped into a common geometric interface\. Point clouds provide explicit spatial information while being easily derived from a wide range of 3D modalities, making them a natural interface for geometry\-conditioned generation\.

Conditioned on this explicit 3D geometry, we formulate brick construction as an autoregressive sequence generation problem\. A key component of BrickAnything is a structure\-aware tree tokenization scheme\. Instead of representing brick assemblies only as spatially ordered sequences, our tokenization organizes bricks according to local attachment relations\. This representation better reflects the physical construction process, produces more compact sequences, and makes structural dependencies between bricks explicit during generation\.

Building on this tokenization, we further propose a buildability\-aware reward that jointly measures structural stability and geometric fidelity\. This reward is used to construct preference pairs for Direct Preference Optimization \(DPO\), enabling the model to optimize buildability\-related objectives that are difficult to capture with likelihood\-based training alone\. During inference, we also introduce a lightweight parent\-aware rollback strategy tailored to the proposed tree tokenization\. When an unstable brick is detected, the generation process rolls back to its parent brick in the tree sequence and regenerates the subsequent structure\. This simple mechanism effectively improves the robustness of autoregressive brick generation\.

Through extensive experiments, we demonstrate that BrickAnything improves both geometric fidelity and buildability compared with conventional brick generation baselines\. In particular, our structure\-aware tokenization explicitly preserves local attachment relations and substantially reduces rollback and regeneration during inference\. We summarize our contributions as follows:

- •We introduceBrickAnything, a unified geometry\-conditioned framework for buildable brick generation\. By using point clouds as a modality\-agnostic intermediate representation, BrickAnything enables explicit 3D\-guided generation from diverse input modalities\.
- •We propose a structure\-aware tree tokenization scheme for brick structures\. Instead of relying only on global spatial ordering, our representation organizes bricks according to local attachment relations, explicitly modeling parent\-child assembly dependencies during autoregressive generation\.
- •We propose a buildability\-aware reward that jointly measures structural stability and geometric fidelity, and use it to construct preference pairs for Direct Preference Optimization \(DPO\), enabling the model to optimize buildability\-related objectives beyond likelihood\-based training\.
- •We design a lightweight parent\-aware rollback strategy tailored to the proposed tree tokenization\. When an unstable brick is detected, the generation process rolls back to its structurally related parent and regenerates the subsequent substructure, improving inference robustness\.

## 2Related work

### 2\.1Brick Structure Generation

Generating brick structures from a reference 3D shape has been widely studied in prior work\[[25](https://arxiv.org/html/2605.26182#bib.bib26),[40](https://arxiv.org/html/2605.26182#bib.bib27)\]\. Early approaches formulate this task as a combinatorial construction problem guided by handcrafted heuristics or search strategies\. These methods typically enforce constraints such as structural connectivity, stability, and efficient brick usage, and often rely on sequential planning or heuristic search to assemble bricks that reconstruct a target geometry\. While effective in producing physically feasible designs, they generally assume that the input shape can be directly realized using discrete bricks\. As a result, they fail when there is no feasible assembly solution for the given brick set — i\.e\., when the target geometry cannot be constructed under the prescribed brick types, connection rules, and stability constraints\. Moreover, the reliance on heuristic rules or search\-based strategies leads to high computational costs and limited scalability\.

Recent learning\-based methods have explored generative brick modeling with autoregressive models\[[32](https://arxiv.org/html/2605.26182#bib.bib22),[19](https://arxiv.org/html/2605.26182#bib.bib24),[13](https://arxiv.org/html/2605.26182#bib.bib23)\]\. BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]generates physically stable and buildable brick structures from text prompts by predicting bricks sequentially, while enforcing validity and physical constraints during inference\. BrickNet\[[19](https://arxiv.org/html/2605.26182#bib.bib24)\]further extends autoregressive brick generation by introducing a graph\-backed connectivity representation and a large\-scale human\-designed LDraw dataset, enabling the modeling of more diverse part types and connection semantics\. LEGO®\-Maker\[[13](https://arxiv.org/html/2605.26182#bib.bib23)\]further explores image\-conditioned LEGO®model generation using an autoregressive framework with multiple brick types, but its experiments mainly focus on a limited set of object categories\. Despite these advances, existing learning\-based methods are mainly conditioned on text descriptions or 2D visual observations, which do not provide explicit and unambiguous 3D geometric constraints\. As a result, they may struggle to faithfully reconstruct complex target geometries or generate structurally valid brick structures that are tightly aligned with an input 3D shape\. This motivates a unified geometry\-conditioned formulation that directly uses 3D shape information while preserving the buildability constraints of brick assemblies\.

### 2\.2Autoregressive Models for 3D Generation

Autoregressive models have become an effective paradigm for modeling structured data by factorizing generation into sequential token prediction\[[43](https://arxiv.org/html/2605.26182#bib.bib28)\]\. Following their success in language\[[33](https://arxiv.org/html/2605.26182#bib.bib29),[1](https://arxiv.org/html/2605.26182#bib.bib30)\]and image generation\[[31](https://arxiv.org/html/2605.26182#bib.bib31),[4](https://arxiv.org/html/2605.26182#bib.bib32),[41](https://arxiv.org/html/2605.26182#bib.bib33)\], this paradigm has also been explored for 3D generation across different representations, including point cloud\[[7](https://arxiv.org/html/2605.26182#bib.bib34),[26](https://arxiv.org/html/2605.26182#bib.bib35)\], Gaussian splatting\[[44](https://arxiv.org/html/2605.26182#bib.bib38),[22](https://arxiv.org/html/2605.26182#bib.bib39)\], and mesh\[[29](https://arxiv.org/html/2605.26182#bib.bib36),[38](https://arxiv.org/html/2605.26182#bib.bib37)\]\. Brick\-based generation can also be naturally formulated as a sequential prediction problem, since a brick assembly of brick structures consists of discrete elements with spatial, topological, and physical dependencies\. Recent methods\[[32](https://arxiv.org/html/2605.26182#bib.bib22),[13](https://arxiv.org/html/2605.26182#bib.bib23),[19](https://arxiv.org/html/2605.26182#bib.bib24)\]serialize brick assemblies into token sequences and generate brick structures step by step\. In this work, we introduce a geometry\-conditioned autoregressive framework that uses point clouds as explicit 3D guidance and organizes brick assemblies according to local attachment relations, enabling more accurate and physically buildable generation\.

### 2\.3RLHF with Direct Preference Optimization

Aligning model outputs with human preferences is a critical step in improving generative models, especially for large language models \(LLMs\)\[[9](https://arxiv.org/html/2605.26182#bib.bib40),[54](https://arxiv.org/html/2605.26182#bib.bib41),[30](https://arxiv.org/html/2605.26182#bib.bib42)\]\. Techniques like reinforcement learning from human feedback \(RLHF\) and direct preference optimization \(DPO\) have become standard for incorporating preference signals\. While early RLHF methods using PPO\[[36](https://arxiv.org/html/2605.26182#bib.bib43)\]often suffer from instability and high computational cost, DPO\[[34](https://arxiv.org/html/2605.26182#bib.bib44)\]eliminates the need for an explicit reward model and directly optimizes policy likelihood\. Recently, preference‑based optimization has also been applied to structured generation tasks, including 3D shape generation\[[49](https://arxiv.org/html/2605.26182#bib.bib3),[53](https://arxiv.org/html/2605.26182#bib.bib45),[14](https://arxiv.org/html/2605.26182#bib.bib46)\]\. In this work, we adapt DPO to brick structure generation by constructing preference pairs with a buildability\-aware reward that jointly considers geometric fidelity and structural stability\.

## 3Method

BrickAnything comprises three components, as illustrated in Figure[1](https://arxiv.org/html/2605.26182#S3.F1)\. Following BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\], we construct brick structures within a20×20×2020\\times 20\\times 20voxel grid using the same eight commonly available standard LEGO®bricks:1×11\\times 1,1×21\\times 2,1×41\\times 4,1×61\\times 6,1×81\\times 8,2×22\\times 2,2×42\\times 4, and2×62\\times 6\. Section[3\.1](https://arxiv.org/html/2605.26182#S3.SS1)introduces our tokenization scheme and model architecture\. Section[3\.2](https://arxiv.org/html/2605.26182#S3.SS2)presents a buildability\-aware DPO post\-training stage, where preference pairs are constructed by jointly considering geometric fidelity and structural stability\. Section[3\.3](https://arxiv.org/html/2605.26182#S3.SS3)introduces validity\-constrained decoding and stability\-guided rollback\.

### 3\.1Structure\-Aware Tokenization and Geometry\-Conditioned Generation

Unlike prior brick generation methods\[[32](https://arxiv.org/html/2605.26182#bib.bib22),[13](https://arxiv.org/html/2605.26182#bib.bib23)\]that represent structures as flattened sequences that preserve spatial order, we propose a structure\-aware tree tokenization scheme that explicitly encodes local attachment relations between bricks\.

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/pipeline.png)Figure 1:BrickAnything Framework Overview\.BrickAnything converts diverse 3D inputs into point clouds as a modality\-agnostic geometric interface\. The framework consists of three stages: 1\)Pre\-training, where a shape encoder and BrickAnything Transformer learn structure\-aware autoregressive brick generation; 2\)Post\-training, where buildability\-aware rewards are used to construct preference pairs and optimize the policy model with reward\-guided DPO; and 3\)Inference, where validity\-constrained decoding and rollback\-based regeneration are applied to produce stable brick structures\.#### Structure\-aware tree tokenization\.

Given a brick structureℬ=\{bi\}i=1N\\mathcal\{B\}=\\\{b\_\{i\}\\\}\_\{i=1\}^\{N\}, each brick is represented asbi=\[hi,wi,xi,yi,zi\]b\_\{i\}=\[h\_\{i\},w\_\{i\},x\_\{i\},y\_\{i\},z\_\{i\}\], where\(hi,wi\)\(h\_\{i\},w\_\{i\}\)denotes its footprint size along thexx\- andyy\-axes, and\(xi,yi,zi\)\(x\_\{i\},y\_\{i\},z\_\{i\}\)is the discrete position of the stud closest to the origin\. We first construct a vertical attachment graph𝒢=\(ℬ,ℰ\)\\mathcal\{G\}=\(\\mathcal\{B\},\\mathcal\{E\}\), where an edge indicates that two bricks are vertically adjacent and overlap in thex​yxy\-plane:

\(bi,bj\)∈ℰ⟺\|zi−zj\|=1​and​Ωix​y∩Ωjx​y≠∅\(b\_\{i\},b\_\{j\}\)\\in\\mathcal\{E\}\\quad\\Longleftrightarrow\\quad\|z\_\{i\}\-z\_\{j\}\|=1\\ \\text\{and\}\\ \\Omega\_\{i\}^\{xy\}\\cap\\Omega\_\{j\}^\{xy\}\\neq\\emptyset\(1\)whereΩix​y\\Omega\_\{i\}^\{xy\}is the footprint ofbib\_\{i\}on thex​yxy\-plane\. For a stable brick structure,𝒢\\mathcal\{G\}is connected\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\]\. We therefore choose the root brick by the lexicographic order of\(z,y,x\)\(z,y,x\)and perform breadth\-first traversal over𝒢\\mathcal\{G\}\. Each unvisited brick is assigned to the first visited neighboring parent, yielding a deterministic BFS spanning tree, as shown in Figure[2](https://arxiv.org/html/2605.26182#S3.F2)\.

The root brick is encoded by its absolute attributes:

\(x0,y0,z0,h0,w0\)\(x\_\{0\},y\_\{0\},z\_\{0\},h\_\{0\},w\_\{0\}\)\(2\)For each non\-root brickbib\_\{i\}, we encode it relative to its parentbp​\(i\)b\_\{p\(i\)\}as

\(fi,hi,wi,mi\)\(f\_\{i\},h\_\{i\},w\_\{i\},m\_\{i\}\)\(3\)wherefif\_\{i\}indexes the attachment position on the parent,\(hi,wi\)\(h\_\{i\},w\_\{i\}\)is the child brick size, andmim\_\{i\}indexes the child\-side anchor position, as shown in Figure[2](https://arxiv.org/html/2605.26182#S3.F2)\. During BFS traversal, children of the same parent are sorted increasingly byfif\_\{i\}\. After all children of the current parent are encoded, anEOPtoken is inserted to mark the end of this local attachment group\. The final sequence is

𝐬=\[BOS,x0,y0,z0,h0,w0,\(f,h,w,m\),…,EOP,\(f,h,w,m\),…,EOP,…,EOS\]\\mathbf\{s\}=\[\\texttt\{BOS\},x\_\{0\},y\_\{0\},z\_\{0\},h\_\{0\},w\_\{0\},\(f,h,w,m\),\\ldots,\\texttt\{EOP\},\(f,h,w,m\),\\ldots,\\texttt\{EOP\},\\ldots,\\texttt\{EOS\}\]\(4\)This representation explicitly attaches each newly generated brick to an existing parent, thereby converting brick generation from global coordinate prediction into local attachment prediction\. More details of the proposed tokenization scheme are provided in Appendix[A](https://arxiv.org/html/2605.26182#A1)\.

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/tokenization.png)Figure 2:Overview of the proposed structure\-aware tree tokenization\. The root brick is encoded by absolute attributes\(x,y,z,h,w\)\(x,y,z,h,w\), while each child brick is encoded by relative attachment tokens\(f,h,w,m\)\(f,h,w,m\)\.EOPmarks the end of each parent’s child group in the BFS traversal\.
#### Shape\-conditioned brick generation\.

Given an input 3D shape, we first convert it into a point cloud representation with surface normals, denoted as𝒫∈ℝN×3\\mathcal\{P\}\\in\\mathbb\{R\}^\{N\\times 3\}and𝒩∈ℝN×3\\mathcal\{N\}\\in\\mathbb\{R\}^\{N\\times 3\}, whereN=8192N=8192\. The point cloud and its normals are encoded by a pre\-trained Michelangelo\[[50](https://arxiv.org/html/2605.26182#bib.bib47)\]encoderℰg\\mathcal\{E\}\_\{g\}, producing a sequence of shape tokens

𝐜g=ℰg​\(𝒫,𝒩\)\\mathbf\{c\}\_\{g\}=\\mathcal\{E\}\_\{g\}\(\\mathcal\{P\},\\mathcal\{N\}\)\(5\)which serves as the geometric condition for brick structure generation\. We choose OPT\-350M\[[48](https://arxiv.org/html/2605.26182#bib.bib48)\]as our autoregressive transformer architecture\. Given the structure\-aware tree sequence𝐬=\(s1,s2,…,sT\)\\mathbf\{s\}=\(s\_\{1\},s\_\{2\},\\ldots,s\_\{T\}\), our transformer decoder with parametersθ\\thetagenerates the sequence conditioned on the shape tokens𝐜g\\mathbf\{c\}\_\{g\}\. The model is trained using the standard next\-token prediction objective:

ℒSFT=−∑i=1Tlog⁡pθ​\(si∣s<i,𝐜g\)\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\sum\_\{i=1\}^\{T\}\\log p\_\{\\theta\}\\left\(s\_\{i\}\\mid s\_\{<i\},\\mathbf\{c\}\_\{g\}\\right\)\(6\)During inference, generation starts from the shape condition and theBOStoken, and the generated sequence is then converted into a brick structure through our tree detokenization algorithm\.

### 3\.2DPO\-based Post\-Training for Buildable Brick Generation

Next\-token prediction optimizes token\-level likelihood but cannot directly enforce non\-differentiable objectives such as geometric fidelity and physical buildability\. To address this, we introduce a reward\-guided DPO post\-training stage\. Specifically, for each input shape, we generate multiple candidate brick structures and rank them according to our proposed buildability\-aware reward\. The ranked candidates are then used to fine\-tune the model with a reward\-weighted DPO objective, together with an auxiliary SFT loss\.

#### Buildability\-aware reward\.

Given an input point cloud𝒫\\mathcal\{P\}and a generated brick structureℬ^\\hat\{\\mathcal\{B\}\}, we evaluate generation quality from two aspects: geometric fidelity and physical stability\. For geometric fidelity, we define a reward that combines voxel\-level occupancy consistency and surface\-level geometric alignment\.

We first voxelize the input point cloud into an occupancy gridV𝒫V\_\{\\mathcal\{P\}\}, and convert the generated brick structure into another occupancy gridVℬ^V\_\{\\hat\{\\mathcal\{B\}\}\}\. The voxel\-level consistency is measured by the intersection\-over\-union:

RIoU=\|V𝒫∩Vℬ^\|\|V𝒫∪Vℬ^\|∈\[0,1\]R\_\{\\mathrm\{IoU\}\}=\\frac\{\|V\_\{\\mathcal\{P\}\}\\cap V\_\{\\hat\{\\mathcal\{B\}\}\}\|\}\{\|V\_\{\\mathcal\{P\}\}\\cup V\_\{\\hat\{\\mathcal\{B\}\}\}\|\}\\in\[0,1\]\(7\)Although IoU measures volumetric overlap, it may overlook fine\-grained surface discrepancies\. Therefore, we introduce a surface\-level distance metric to complement the voxel\-level score and better capture boundary alignment and local geometric details\.

Specifically, we extract a surface meshℳ^\\hat\{\\mathcal\{M\}\}from the generated voxel gridVℬ^V\_\{\\hat\{\\mathcal\{B\}\}\}using Marching Cubes\[[45](https://arxiv.org/html/2605.26182#bib.bib49)\], and uniformly sample a point cloud𝒫^\\hat\{\\mathcal\{P\}\}from the reconstructed surface\. Before computing the distance, both𝒫^\\hat\{\\mathcal\{P\}\}and𝒫\\mathcal\{P\}are centered at their centroids and scaled by their maximum radial distances\. We then compute the Chamfer Distance between the normalized point clouds:

DCD​\(𝒫,𝒫^\)=1\|𝒫\|​∑𝐱∈𝒫min𝐱^∈𝒫^⁡‖𝐱−𝐱^‖2\+1\|𝒫^\|​∑𝐱^∈𝒫^min𝐱∈𝒫⁡‖𝐱^−𝐱‖2D\_\{\\mathrm\{CD\}\}\(\\mathcal\{P\},\\hat\{\\mathcal\{P\}\}\)=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{P\}\}\\min\_\{\\hat\{\\mathbf\{x\}\}\\in\\hat\{\\mathcal\{P\}\}\}\\\|\\mathbf\{x\}\-\\hat\{\\mathbf\{x\}\}\\\|\_\{2\}\+\\frac\{1\}\{\|\\hat\{\\mathcal\{P\}\}\|\}\\sum\_\{\\hat\{\\mathbf\{x\}\}\\in\\hat\{\\mathcal\{P\}\}\}\\min\_\{\\mathbf\{x\}\\in\\mathcal\{P\}\}\\\|\\hat\{\\mathbf\{x\}\}\-\\mathbf\{x\}\\\|\_\{2\}\(8\)A smaller Chamfer Distance indicates better surface\-level alignment between the generated brick structure and the target geometry\. We convert this distance into a bounded reward by

RCD=max⁡\(1−5​DCD​\(𝒫,𝒫^\),0\)∈\[0,1\]R\_\{\\mathrm\{CD\}\}=\\max\\left\(1\-5D\_\{\\mathrm\{CD\}\}\(\\mathcal\{P\},\\hat\{\\mathcal\{P\}\}\),0\\right\)\\in\[0,1\]\(9\)The final geometric reward is defined as the sum of the voxel\-level and surface\-level rewards:

Rgeo=RIoU\+RCD∈\[0,2\]R\_\{\\mathrm\{geo\}\}=R\_\{\\mathrm\{IoU\}\}\+R\_\{\\mathrm\{CD\}\}\\in\[0,2\]\(10\)For structural stability, we follow StableLego\[[24](https://arxiv.org/html/2605.26182#bib.bib50)\]and estimate a per\-brick stability scoresi∈\[0,1\]s\_\{i\}\\in\[0,1\]based on static equilibrium under frictional constraints\. Since a single unstable brick can compromise the whole assembly, we define the stability reward as the minimum per\-brick score:

Rstable=minbi∈ℬ^⁡si∈\[0,1\]R\_\{\\mathrm\{stable\}\}=\\min\_\{b\_\{i\}\\in\\hat\{\\mathcal\{B\}\}\}s\_\{i\}\\in\[0,1\]\(11\)The overall reward is therefore

R​\(ℬ^,𝒫\)=Rgeo\+Rstable∈\[0,3\]R\(\\hat\{\\mathcal\{B\}\},\\mathcal\{P\}\)=R\_\{\\mathrm\{geo\}\}\+R\_\{\\mathrm\{stable\}\}\\in\[0,3\]\(12\)
![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/dpo_set.png)Figure 3:Examples of the collected preference pairs for reward\-guided DPO\.
#### Preference pair construction\.

For each input shape, we sample six legal candidate brick structures from the SFT model and compute their rewards\. We construct pairwise comparisons and retain only informative pairs whose reward gap is at least0\.20\.2and whose higher reward is no less than11\. For each retained pair, the higher\-reward candidate is used as the preferred sampleywy\_\{w\}, and the other as the rejected sampleyly\_\{l\}\. Figure[3](https://arxiv.org/html/2605.26182#S3.F3)shows some selection cases of our collected preference pairs\.

#### Reward\-guided DPO\.

Given an input conditionxx, a preferred sequenceywy\_\{w\}, and a rejected sequenceyly\_\{l\}, we optimize the policy modelπθ\\pi\_\{\\theta\}with a reward\-weighted DPO objective:

ℒDPO=−𝔼\(x,yw,yl\)​\[Δ​R​log⁡σ​\(β​log⁡πθ​\(yw\|x\)​πref​\(yl\|x\)πref​\(yw\|x\)​πθ​\(yl\|x\)\)\]\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\left\[\\Delta R\\log\\sigma\\left\(\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\|x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\|x\)\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\}\\right\)\\right\]\(13\)whereπref\\pi\_\{\\mathrm\{ref\}\}is the frozen reference model,β\\betacontrols the strength of the KL constraint, and

Δ​R=R​\(yw,x\)−R​\(yl,x\)\\Delta R=R\(y\_\{w\},x\)\-R\(y\_\{l\},x\)\(14\)The reward gapΔ​R\\Delta Rassigns larger weights to pairs with clearer quality differences, encouraging the model to distinguish high\-fidelity and stable assemblies from inferior ones\.

To preserve the original data distribution, we further add an auxiliary SFT loss on the ground\-truth sequenceygty\_\{\\mathrm\{gt\}\}:

ℒSFT=−∑t=1Tlog⁡πθ​\(ygt,t∣ygt,<t,x\)\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\sum\_\{t=1\}^\{T\}\\log\\pi\_\{\\theta\}\(y\_\{\\mathrm\{gt\},t\}\\mid y\_\{\\mathrm\{gt\},<t\},x\)\(15\)The final post\-training objective is

ℒpost=ℒDPO\+λ​ℒSFT\\mathcal\{L\}\_\{\\mathrm\{post\}\}=\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(16\)whereλ\\lambdabalances preference alignment and distribution preservation\.

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/ans.png)Figure 4:Qualitative comparison\.BrickAnything produces stable and geometry\-faithful structures than the baselines\. Red boxes mark Legolization failures\.

### 3\.3Validity\-Constrained Decoding and Stability\-Guided Rollback

#### Validity\-constrained decoding\.

Autoregressive decoding may produce invalid local decisions, including unsupported brick sizes, invalid attachment tokens, and spatial collisions\. Instead of accepting each decoded child tuple directly, we perform tuple\-level validity checking during inference\. For each child brick, the model samples a candidate tuple\(f,h,w,m\)\(f,h,w,m\), which is decoded into a concrete brick placement based on the current parent brick and BFS traversal state\. The candidate tuple is then verified in two stages\. First, we check whether the token combination defines a valid attachment, including a feasible parent connectorff, a brick size\(h,w\)\(h,w\)from the predefined brick library, and a valid child\-side anchormm\. Second, after the candidate is converted into grid occupancy, we check whether it overlaps with any previously generated brick\. If either check fails, the candidate tuple is rejected and resampled\. The accepted tuple is appended to the sequence only after passing these validity checks\.

#### stability\-guided rollback\.

Although validity\-constrained decoding filters invalid local decisions, it does not guarantee the global physical stability of the complete structure\. To further improve physical buildability, we perform stability\-guided rollback after a complete brick structure is generated\. Following StableLego\[[24](https://arxiv.org/html/2605.26182#bib.bib50)\], we compute a per\-brick stability scoresi∈\[0,1\]s\_\{i\}\\in\[0,1\]for each brick in the generated structure, wheresi=0s\_\{i\}=0denotes that brickbib\_\{i\}is unstable\. If the generated structure contains unstable bricks, we identify the first unstable brick according to the generation order:

k=min⁡\{i∣si=0\}k=\\min\\\{i\\mid s\_\{i\}=0\\\}\(17\)We then trace its parent brickbp​\(k\)b\_\{p\(k\)\}in the generated tree structure and locate the token position wherebp​\(k\)b\_\{p\(k\)\}itself is generated in the generated token sequence\. The sequence is rolled back to the state before generatingbp​\(k\)b\_\{p\(k\)\}and the subsequent tokens are regenerated\. This process is repeated until a stable structure is obtained or the maximum rollback budget is reached\. The parent\-child attachment relations allow our strategy to locate the source of instability more precisely, regenerate the affected substructure, and preserve stable preceding bricks, leading to more targeted and structurally consistent correction\.

Table 1:Quantitative comparison with baselines and ablations\.We compare BrickAnything with representative baselines and ablated variants on the challenging and stable subsets\. Lower CD and rollback averages are better, while higher IoU, %Stable, and %Valid indicate better performance\.,, andindicate the best, second\-best, and third\-best results, respectively\.\(a\) Challenging subset

MethodCD↓\\downarrowIoU↑\\uparrowRollback↓\\downarrow%Stable↑\\uparrow%Valid↑\\uparrowLegolization\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\]–––0\.0%\\cellcolorbest100%BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]\-style tokenization w/o DPO0\.13070\.5446\.99174\.0%\\cellcolorbest100%BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]\-style tokenization0\.13090\.553\\cellcolorthird6\.750\\cellcolorthird76\.0%\\cellcolorbest100%Ours w/o Validity\-constrained decoding or Rollback or DPO\\cellcolorbest0\.1271\\cellcolorbest0\.602–60\.6%68\.6%Ours w/o Rollback or DPO\\cellcolorsecond0\.1283\\cellcolorsecond0\.597–72\.0%\\cellcolorbest100%Ours w/o DPO0\.13010\.573\\cellcolorsecond0\.444\\cellcolorsecond82\.4%\\cellcolorbest100%Ours\\cellcolorthird0\.1299\\cellcolorthird0\.586\\cellcolorbest0\.422\\cellcolorbest83\.4%\\cellcolorbest100%

\(b\) Stable subset

MethodCD↓\\downarrowIoU↑\\uparrowRollback↓\\downarrow%Stable↑\\uparrow%Valid↑\\uparrowLegolization\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\]\\cellcolorbest0\.1120\\cellcolorbest1\.000–\\cellcolorbest100%\\cellcolorbest100%BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]\-style tokenization w/o DPO0\.12970\.7213\.916\\cellcolorbest100%\\cellcolorbest100%BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]\-style tokenization0\.12920\.7423\.620\\cellcolorbest100%\\cellcolorbest100%Ours w/o Validity\-constrained decoding or Rollback or DPO\\cellcolorsecond0\.1265\\cellcolorsecond0\.798–63\.4%71\.6%Ours w/o Rollback or DPO\\cellcolorthird0\.12740\.772–91\.0%\\cellcolorbest100%Ours w/o DPO0\.12900\.751\\cellcolorsecond0\.216\\cellcolorbest100%\\cellcolorbest100%Ours0\.1283\\cellcolorthird0\.788\\cellcolorbest0\.184\\cellcolorbest100%\\cellcolorbest100%

## 4Experiments

### 4\.1Implementation Details

#### Dataset\.

We construct our training data from a curated collection of approximately 230K high\-quality meshes from ShapeNet\[[2](https://arxiv.org/html/2605.26182#bib.bib52)\], Objaverse\[[11](https://arxiv.org/html/2605.26182#bib.bib53)\], and Objaverse\-XL\[[10](https://arxiv.org/html/2605.26182#bib.bib54)\], following recent mesh curation practice\[[6](https://arxiv.org/html/2605.26182#bib.bib51)\]\. Applying Legolization\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\]yields around 168K stable mesh–brick pairs, of which 4K are randomly selected for validation and the rest are used for training\. For evaluation, we build two test subsets: a stable subset with 500 validation samples successfully converted by Legolization, and a challenging subset with 500 meshes sampled from cases where Legolization fails to produce stable structures\. More training details are provided in Appendix[C](https://arxiv.org/html/2605.26182#A3)\.

### 4\.2Metrics and Baselines

#### Metrics

We evaluate generated brick structures in terms of geometric fidelity, structural validity, physical stability, and inference correction cost\. Geometric fidelity is measured by Voxel IoU and Chamfer Distance, as defined in Sec\.[3\.2](https://arxiv.org/html/2605.26182#S3.SS2.SSS0.Px1); structural validity is the percentage of outputs satisfying valid brick types, workspace bounds, and collision\-free placement; physical stability is measured by the stable generation rate following StableLego\[[24](https://arxiv.org/html/2605.26182#bib.bib50)\]; and inference cost is measured by the average number of rollback operations\. Unless otherwise specified, Voxel IoU, Chamfer Distance, and rollback average are computed over final generated samples that are physically stable\.

#### Baselines\.

We compare our method with two representative baselines\. First, we implement a baseline using the same point\-cloud encoder and transformer architecture as ours, but replacing the proposed structure\-aware tree tokenization with a conventional BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]\-style tokenization\. Specifically, each brick is represented as\(h,w,x,y,z\)\(h,w,x,y,z\), and bricks are sorted into a flat sequence by the lexicographic order of\(z,y,x\)\(z,y,x\)\. The baseline is pretrained on the same dataset as our model and equipped with the same brick\-by\-brick rejection sampling and physics\-aware rollback strategies used in BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\], isolating the effect of the proposed tokenization scheme\. Second, we compare with Legolization\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\], a heuristic search\-based method for converting 3D shapes into stable brick structures\. This comparison evaluates whether our learned model can produce geometrically faithful and physically buildable structures, especially in cases where heuristic search fails\.

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/abladpo.png)Figure 5:Qualitative ablation results\.Each component improves buildability or fidelity: validity\-constrained decoding removes invalid placements, rollback corrects instability, and DPO enhances shape faithfulness\.

### 4\.3Comparison

Table[1](https://arxiv.org/html/2605.26182#S3.T1)compares BrickAnything with representative baselines on the challenging and stable subsets\. On the challenging subset, Legolization\[[25](https://arxiv.org/html/2605.26182#bib.bib26)\]obtains0\.0%0\.0\\%stable rates, indicating that heuristic search is brittle for geometrically difficult targets\. The BrickGPT\-style baseline\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]benefits from DPO, improving the stable rate from74\.0%74\.0\\%to76\.0%76\.0\\%, IoU from0\.5440\.544to0\.5530\.553, and reducing the average number of rollbacks from6\.9916\.991to6\.7506\.750\. In contrast, BrickAnything achieves stronger buildability and fidelity, reaching83\.4%83\.4\\%stable rate,100%100\\%valid rate,0\.5860\.586IoU, and a much lower rollback average of0\.4220\.422\.

On the stable subset, Legolization obtains the best CD and IoU because this subset is selected from its successful cases\. Among learned methods, BrickAnything achieves the best overall performance\. Compared with the BrickGPT\-style baseline, BrickAnything improves IoU from0\.7420\.742to0\.7880\.788, reduces CD from0\.12920\.1292to0\.12830\.1283, and lowers the rollback average from3\.6203\.620to0\.1840\.184, while maintaining100%100\\%stable and valid rates\. Together with Figure[4](https://arxiv.org/html/2605.26182#S3.F4), these results show that structure\-aware tree tokenization improves geometric fidelity and sequence coherence, while DPO, validity\-constrained decoding, and rollback enhance buildability with fewer correction steps\. Additional qualitative results, failure analyses, and image\- and text\-to\-brick extensions are provided in Appendix[B](https://arxiv.org/html/2605.26182#A2)\.

### 4\.4Ablation Studies

Table[1](https://arxiv.org/html/2605.26182#S3.T1)and Figure[5](https://arxiv.org/html/2605.26182#S4.F5)ablate the main components of BrickAnything\. Removing validity\-constrained decoding, rollback, and DPO leads to strong geometric scores but substantially weaker buildability, with the stable and valid rates dropping to60\.6%60\.6\\%and68\.6%68\.6\\%on the challenging subset\. Adding validity\-constrained decoding restores the valid rate to100%100\\%, but the stable rate remains limited without rollback, indicating that local validity constraints alone cannot guarantee global physical stability\. Rollback further improves stability by correcting unstable generations through structurally related resampling\. Finally, DPO improves the generation distribution itself: compared with the variant without DPO, the full model improves IoU from0\.5730\.573to0\.5860\.586on the challenging subset and from0\.7510\.751to0\.7880\.788on the stable subset, while reducing the average number of rollbacks from0\.4440\.444to0\.4220\.422and from0\.2160\.216to0\.1840\.184, respectively\. These results show that validity\-constrained decoding, rollback, and DPO address complementary aspects of buildable brick generation\. We provide a detailed ablation analysis in Appendix[D](https://arxiv.org/html/2605.26182#A4)\.

## 5Conclusion

We presentedBrickAnything, a geometry\-conditioned framework for buildable brick generation\. By combining point\-cloud conditioning with structure\-aware tree tokenization, BrickAnything models local attachment relations in autoregressive generation\. Buildability\-aware DPO, validity\-constrained decoding, and stability\-guided rollback further improve fidelity and stability\. Experiments demonstrate superior performance over coordinate\-ordered tokenization baselines and robustness when heuristic Legolization fails\. We discuss the limitations of our work and potential directions for future research in Appendix[E](https://arxiv.org/html/2605.26182#A5)\.

## References

- \[1\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[2\]A\. X\. Chang, T\. Funkhouser, L\. Guibas, P\. Hanrahan, Q\. Huang, Z\. Li, S\. Savarese, M\. Savva, S\. Song, H\. Su,et al\.\(2015\)Shapenet: an information\-rich 3d model repository\.arXiv preprint arXiv:1512\.03012\.Cited by:[§4\.1](https://arxiv.org/html/2605.26182#S4.SS1.SSS0.Px1.p1.1)\.
- \[3\]\(2024\-06\)PixelSplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19457–19467\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[4\]M\. Chen, A\. Radford, R\. Child, J\. Wu, H\. Jun, D\. Luan, and I\. Sutskever\(2020\)Generative pretraining from pixels\.InInternational conference on machine learning,pp\. 1691–1703\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[5\]Y\. Chen, T\. He, D\. Huang, W\. Ye, S\. Chen, J\. Tang, X\. Chen, Z\. Cai, L\. Yang, G\. Yu,et al\.\(2024\)Meshanything: artist\-created mesh generation with autoregressive transformers\.arXiv preprint arXiv:2406\.10163\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[6\]Y\. Chen, Y\. Wang, Y\. Luo, Z\. Wang, Z\. Chen, J\. Zhu, C\. Zhang, and G\. Lin\(2025\)Meshanything v2: artist\-created mesh generation with adjacent mesh tokenization\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 13922–13931\.Cited by:[§4\.1](https://arxiv.org/html/2605.26182#S4.SS1.SSS0.Px1.p1.1)\.
- \[7\]A\. Cheng, X\. Li, S\. Liu, M\. Sun, and M\. Yang\(2022\)Autoregressive 3d shape generation via canonical mapping\.InEuropean Conference on Computer Vision,pp\. 89–104\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[8\]Y\. Cheng, H\. Lee, S\. Tulyakov, A\. G\. Schwing, and L\. Gui\(2023\-06\)SDFusion: multimodal 3d shape completion, reconstruction, and generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 4456–4465\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[9\]P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei\(2017\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[10\]M\. Deitke, R\. Liu, M\. Wallingford, H\. Ngo, O\. Michel, A\. Kusupati, A\. Fan, C\. Laforte, V\. Voleti, S\. Y\. Gadre,et al\.\(2023\)Objaverse\-xl: a universe of 10m\+ 3d objects\.Advances in neural information processing systems36,pp\. 35799–35813\.Cited by:[§4\.1](https://arxiv.org/html/2605.26182#S4.SS1.SSS0.Px1.p1.1)\.
- \[11\]M\. Deitke, D\. Schwenk, J\. Salvador, L\. Weihs, O\. Michel, E\. VanderBilt, L\. Schmidt, K\. Ehsani, A\. Kembhavi, and A\. Farhadi\(2023\)Objaverse: a universe of annotated 3d objects\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13142–13153\.Cited by:[§4\.1](https://arxiv.org/html/2605.26182#S4.SS1.SSS0.Px1.p1.1)\.
- \[12\]Z\. Fan, K\. Wang, K\. Wen, Z\. Zhu, D\. Xu, and Z\. Wang\(2024\)Lightgaussian: unbounded 3d gaussian compression with 15x reduction and 200\+ fps\.Advances in neural information processing systems37,pp\. 140138–140158\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[13\]J\. Ge, M\. Zhou, H\. Zheng, H\. Xu, and C\. Fu\(2025\)LEGO®\-maker: autoregressive image\-conditioned lego® model creation\.ACM Transactions on Graphics \(TOG\)44\(6\),pp\. 1–15\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.26182#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2605.26182#S3.SS1.p1.1)\.
- \[14\]J\. Guo, J\. Liu, J\. Chen, S\. Mao, C\. Hu, P\. Jiang, J\. Yu, J\. Xu, Q\. Liu, L\. Xu,et al\.\(2025\)Auto\-connect: connectivity\-preserving rigformer with direct preference optimization\.arXiv preprint arXiv:2506\.11430\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[15\]Z\. Huang, J\. Johnson, S\. Debnath, J\. M\. Rehg, and C\. Wu\(2024\)Pointinfinity: resolution\-invariant point diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10050–10060\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[16\]K\. Hui, C\. Liu, X\. Zeng, C\. Fu, and A\. Vahdat\(2025\)Not\-so\-optimal transport flows for 3d point cloud generation\.arXiv preprint arXiv:2502\.12456\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[17\]L\. Jiang, Y\. Mao, L\. Xu, T\. Lu, K\. Ren, Y\. Jin, X\. Xu, M\. Yu, J\. Pang, F\. Zhao, D\. Lin, and B\. Dai\(2025\-12\)AnySplat: feed\-forward 3d gaussian splatting from unconstrained views\.ACM Transactions on Graphics \(TOG\)44\(6\),pp\. 1–16\.External Links:[Document](https://dx.doi.org/10.1145/3763326)Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[18\]B\. Kerbl, G\. Kopanas, T\. Leimkühler, G\. Drettakis,et al\.\(2023\)3d gaussian splatting for real\-time radiance field rendering\.\.ACM Transactions on Graphics \(TOG\)42\(4\),pp\. 139–1\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[19\]P\. Kulits and C\. Schmid\(2026\)BrickNet: graph\-backed generative brick assembly\.arXiv preprint arXiv:2604\.22984\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.26182#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[20\]Z\. Lai, Y\. Zhao, H\. Liu, Z\. Zhao, Q\. Lin, H\. Shi, X\. Yang, M\. Yang, S\. Yang, Y\. Feng,et al\.\(2025\)Hunyuan3d 2\.5: towards high\-fidelity 3d assets generation with ultimate details\.arXiv preprint arXiv:2506\.16504\.Cited by:[Figure 10](https://arxiv.org/html/2605.26182#A2.F10),[Figure 9](https://arxiv.org/html/2605.26182#A2.F9),[§B\.3](https://arxiv.org/html/2605.26182#A2.SS3.p1.1)\.
- \[21\]K\. Lennon, K\. Fransen, A\. O’Brien, Y\. Cao, M\. Beveridge, Y\. Arefeen, N\. Singh, and I\. Drori\(2021\)Image2lego: customized lego set generation from images\.arXiv preprint arXiv:2108\.08477\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p2.1)\.
- \[22\]H\. Liu, X\. Wang, Y\. Wang, Z\. Wu, Z\. Wan, Y\. Ma, R\. Liu, B\. Zhou, Y\. Shen, and Q\. Chen\(2026\)AvatarPointillist: autoregressive 4d gaussian avatarization\.arXiv preprint arXiv:2604\.04787\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[23\]J\. Liu, J\. Xu, S\. Guo, J\. Li, J\. Guo, J\. Yu, H\. Weng, B\. Lei, X\. Yang, Z\. Chen,et al\.\(2025\)Mesh\-rft: enhancing mesh generation via fine\-grained reinforcement fine\-tuning\.arXiv preprint arXiv:2505\.16761\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[24\]R\. Liu, K\. Deng, Z\. Wang, and C\. Liu\(2024\)Stablelego: stability analysis of block stacking assembly\.IEEE Robotics and Automation Letters9\(11\),pp\. 9383–9390\.Cited by:[§3\.2](https://arxiv.org/html/2605.26182#S3.SS2.SSS0.Px1.p3.6),[§3\.3](https://arxiv.org/html/2605.26182#S3.SS3.SSS0.Px2.p1.3),[§4\.2](https://arxiv.org/html/2605.26182#S4.SS2.SSS0.Px1.p1.1)\.
- \[25\]S\. Luo, Y\. Yue, C\. Huang, Y\. Chung, S\. Imai, T\. Nishita, and B\. Chen\(2015\)Legolization: optimizing lego designs\.ACM Transactions on Graphics \(ToG\)34\(6\),pp\. 1–12\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.26182#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2605.26182#S3.SS1.SSS0.Px1.p1.14),[Table 1](https://arxiv.org/html/2605.26182#S3.T1.10.5.6.1),[Table 1](https://arxiv.org/html/2605.26182#S3.T1.5.5.6.1),[§4\.1](https://arxiv.org/html/2605.26182#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.26182#S4.SS2.SSS0.Px2.p1.2),[§4\.3](https://arxiv.org/html/2605.26182#S4.SS3.p1.11)\.
- \[26\]Z\. Meng, Q\. Wang, Z\. Zhou, I\. King, and P\. Zhao\(2025\)3D point cloud generation via autoregressive up\-sampling\.arXiv e\-prints,pp\. arXiv–2503\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[27\]B\. Mildenhall, P\. P\. Srinivasan, M\. Tancik, J\. T\. Barron, R\. Ramamoorthi, and R\. Ng\(2021\)Nerf: representing scenes as neural radiance fields for view synthesis\.Communications of the ACM65\(1\),pp\. 99–106\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[28\]P\. Mittal, Y\. Cheng, M\. Singh, and S\. Tulsiani\(2022\-06\)AutoSDF: shape priors for 3d completion, reconstruction and generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 306–315\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[29\]C\. Nash, Y\. Ganin, S\. A\. Eslami, and P\. Battaglia\(2020\)Polygen: an autoregressive generative model of 3d meshes\.InInternational conference on machine learning,pp\. 7220–7229\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[30\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[31\]N\. Parmar, A\. Vaswani, J\. Uszkoreit, L\. Kaiser, N\. Shazeer, A\. Ku, and D\. Tran\(2018\)Image transformer\.InInternational conference on machine learning,pp\. 4055–4064\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[32\]A\. Pun, K\. Deng, R\. Liu, D\. Ramanan, C\. Liu, and J\. Zhu\(2025\)Generating physically stable and buildable brick structures from text\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 14798–14809\.Cited by:[§A\.3](https://arxiv.org/html/2605.26182#A1.SS3.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2605.26182#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.26182#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2605.26182#S3.SS1.p1.1),[Table 1](https://arxiv.org/html/2605.26182#S3.T1.10.5.7.1),[Table 1](https://arxiv.org/html/2605.26182#S3.T1.10.5.8.1),[Table 1](https://arxiv.org/html/2605.26182#S3.T1.5.5.7.1),[Table 1](https://arxiv.org/html/2605.26182#S3.T1.5.5.8.1),[§3](https://arxiv.org/html/2605.26182#S3.p1.9),[§4\.2](https://arxiv.org/html/2605.26182#S4.SS2.SSS0.Px2.p1.2),[§4\.3](https://arxiv.org/html/2605.26182#S4.SS3.p1.11)\.
- \[33\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[34\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[35\]Z\. Ren, M\. Kim, F\. Liu, and X\. Liu\(2024\)Tiger: time\-varying denoising model for 3d point cloud generation with diffusion process\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9462–9471\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[36\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[37\]E\. Sella, N\. Atia, R\. Mokady, and H\. Averbuch\-Elor\(2025\)Blended point cloud diffusion for localized text\-guided shape editing\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 19119–19129\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[38\]Y\. Siddiqui, A\. Alliegro, A\. Artemov, T\. Tommasi, D\. Sirigatti, V\. Rosov, A\. Dai, and M\. Nießner\(2024\)Meshgpt: generating triangle meshes with decoder\-only transformers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19615–19625\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[39\]J\. Tang, Z\. Li, Z\. Hao, X\. Liu, G\. Zeng, M\. Liu, and Q\. Zhang\(2024\)Edgerunner: auto\-regressive auto\-encoder for artistic mesh generation\.arXiv preprint arXiv:2409\.18114\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[40\]R\. P\. Testuz, Y\. Schwartzburg, and M\. Pauly\(2013\)Automatic generation of constructable brick sculptures\.InEUROGRAPHICS 2013,Cited by:[§2\.1](https://arxiv.org/html/2605.26182#S2.SS1.p1.1)\.
- \[41\]K\. Tian, Y\. Jiang, Z\. Yuan, B\. Peng, and L\. Wang\(2024\)Visual autoregressive modeling: scalable image generation via next\-scale prediction\.Advances in neural information processing systems37,pp\. 84839–84865\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[42\]A\. Vahdat, F\. Williams, Z\. Gojcic, O\. Litany, S\. Fidler, K\. Kreis,et al\.\(2022\)Lion: latent point diffusion models for 3d shape generation\.Advances in neural information processing systems35,pp\. 10021–10039\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[43\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[44\]N\. von Lützow, B\. Rössle, K\. Schmid, and M\. Nießner\(2026\)GaussianGPT: towards autoregressive 3d gaussian scene generation\.arXiv preprint arXiv:2603\.26661\.Cited by:[§2\.2](https://arxiv.org/html/2605.26182#S2.SS2.p1.1)\.
- \[45\]L\. WE\(1987\)Marching cubes: a high resolution 3d surface construction algorithm\.Computer graphics21\(1\),pp\. 7–12\.Cited by:[§3\.2](https://arxiv.org/html/2605.26182#S3.SS2.SSS0.Px1.p3.5)\.
- \[46\]S\. Wu, Y\. Lin, F\. Zhang, Y\. Zeng, Y\. Yang, Y\. Bao, J\. Qian, S\. Zhu, X\. Cao, P\. Torr,et al\.\(2025\)Direct3d\-s2: gigascale 3d generation made easy with spatial sparse attention\.arXiv preprint arXiv:2505\.17412\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[47\]X\. Yan, L\. Lin, N\. J\. Mitra, D\. Lischinski, D\. Cohen\-Or, and H\. Huang\(2022\-06\)ShapeFormer: transformer\-based shape completion via sparse representation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 6239–6249\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[48\]S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)Opt: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§3\.1](https://arxiv.org/html/2605.26182#S3.SS1.SSS0.Px2.p1.7)\.
- \[49\]R\. Zhao, J\. Ye, Z\. Wang, G\. Liu, Y\. Chen, Y\. Wang, and J\. Zhu\(2025\)Deepmesh: auto\-regressive artist\-mesh creation with reinforcement learning\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 10612–10623\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1),[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[50\]Z\. Zhao, W\. Liu, X\. Chen, X\. Zeng, R\. Wang, P\. Cheng, B\. Fu, T\. Chen, G\. Yu, and S\. Gao\(2023\)Michelangelo: conditional 3d shape generation based on shape\-image\-text aligned latent representation\.Advances in neural information processing systems36,pp\. 73969–73982\.Cited by:[§3\.1](https://arxiv.org/html/2605.26182#S3.SS1.SSS0.Px2.p1.4)\.
- \[51\]C\. Zhou, F\. Zhong, P\. Hanji, Z\. Guo, K\. Fogarty, A\. Sztrajman, H\. Gao, and C\. Oztireli\(2024\)Frepolad: frequency\-rectified point latent diffusion for point cloud generation\.InEuropean Conference on Computer Vision,pp\. 434–453\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[52\]Z\. Zhou, J\. Liu, B\. Lei, J\. Xu, H\. Weng, Y\. Zhu, Z\. Chen, J\. Fan, Y\. Ma, D\. Du,et al\.\(2026\)Mesh\-pro: asynchronous advantage\-guided ranking preference optimization for artist\-style quadrilateral mesh generation\.arXiv preprint arXiv:2603\.00526\.Cited by:[§1](https://arxiv.org/html/2605.26182#S1.p1.1)\.
- \[53\]Z\. Zhou, X\. Xia, F\. Ma, H\. Fan, Y\. Yang, and T\. Chua\(2025\)Dreamdpo: aligning text\-to\-3d generation with human preferences via direct preference optimization\.arXiv preprint arXiv:2502\.04370\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.
- \[54\]D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving\(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[§2\.3](https://arxiv.org/html/2605.26182#S2.SS3.p1.1)\.

## Appendix

## Appendix AMore Details of Structure\-Aware Tree Tokenization

### A\.1Attachment Token Definition and Detokenization

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/attachment2.png)Figure 6:Example of attachment token definition\.The selected shared stud is expressed in the local coordinate systems of both the parent and child bricks\. The parent\-side local coordinate determinesfif\_\{i\}, while the child\-side local coordinate determinesmim\_\{i\}\.For each non\-root brickbib\_\{i\}, we encode its attachment to the parent brickbp​\(i\)b\_\{p\(i\)\}using a deterministic reference stud within their overlapping footprint\. LetΩp​\(i\)x​y\\Omega\_\{p\(i\)\}^\{xy\}andΩix​y\\Omega\_\{i\}^\{xy\}denote the sets of occupied stud locations of the parent and child bricks in the globalx​yxy\-plane\. Since a valid vertical attachment requiresΩp​\(i\)x​y∩Ωix​y≠∅\\Omega\_\{p\(i\)\}^\{xy\}\\cap\\Omega\_\{i\}^\{xy\}\\neq\\emptyset, we select a canonical attachment location\(qix,qiy\)=min⁡\(Ωp​\(i\)x​y∩Ωix​y\)\(q\_\{i\}^\{x\},q\_\{i\}^\{y\}\)=\\min\\left\(\\Omega\_\{p\(i\)\}^\{xy\}\\cap\\Omega\_\{i\}^\{xy\}\\right\)wheremin\\mindenotes the lexicographically smallest stud location\. This deterministic choice removes ambiguity when multiple studs overlap\. We then express this global attachment location in the local coordinate systems of the parent and child bricks:

\(uip,vip\)=\(qix−xp​\(i\),qiy−yp​\(i\)\)\(u\_\{i\}^\{p\},v\_\{i\}^\{p\}\)=\(q\_\{i\}^\{x\}\-x\_\{p\(i\)\},\\,q\_\{i\}^\{y\}\-y\_\{p\(i\)\}\)\(18\)\(uic,vic\)=\(qix−xi,qiy−yi\)\(u\_\{i\}^\{c\},v\_\{i\}^\{c\}\)=\(q\_\{i\}^\{x\}\-x\_\{i\},\\,q\_\{i\}^\{y\}\-y\_\{i\}\)\(19\)
The parent\-side attachment tokenfif\_\{i\}encodes the vertical direction and the local attachment position on the parent brick, while the child\-side anchor tokenmim\_\{i\}encodes the corresponding local anchor position on the child brick:

fi=si​hp​\(i\)​wp​\(i\)\+vip​hp​\(i\)\+uipf\_\{i\}=s\_\{i\}h\_\{p\(i\)\}w\_\{p\(i\)\}\+v\_\{i\}^\{p\}h\_\{p\(i\)\}\+u\_\{i\}^\{p\}\(20\)mi=vic​hi\+uicm\_\{i\}=v\_\{i\}^\{c\}h\_\{i\}\+u\_\{i\}^\{c\}\(21\)wheresi∈\{0,1\}s\_\{i\}\\in\\\{0,1\\\}indicates the vertical attachment direction, withsi=0s\_\{i\}=0meaning that the child is attached above the parent andsi=1s\_\{i\}=1meaning that it is attached below the parent\.

Figure[6](https://arxiv.org/html/2605.26182#A1.F6)shows a simple example\. The selected shared stud has local coordinate\(uip,vip\)=\(1,0\)\(u\_\{i\}^\{p\},v\_\{i\}^\{p\}\)=\(1,0\)on the parent brick and\(uic,vic\)=\(0,0\)\(u\_\{i\}^\{c\},v\_\{i\}^\{c\}\)=\(0,0\)on the child brick\. Since the child is attached above the parent, we havesi=0s\_\{i\}=0\. For a parent brick with\(hp​\(i\),wp​\(i\)\)=\(4,2\)\(h\_\{p\(i\)\},w\_\{p\(i\)\}\)=\(4,2\)and a child brick withhi=2h\_\{i\}=2, the corresponding tokens arefi=0×2×4\+0×4\+1=1f\_\{i\}=0\\times 2\\times 4\+0\\times 4\+1=1andmi=0×2\+0=0m\_\{i\}=0\\times 2\+0=0\.

During detokenization, given the parent state and the child size\(hi,wi\)\(h\_\{i\},w\_\{i\}\), we first recover the local attachment variables fromfif\_\{i\}andmim\_\{i\}:si=⌊fi/\(hp​\(i\)​wp​\(i\)\)⌋s\_\{i\}=\\left\\lfloor f\_\{i\}/\(h\_\{p\(i\)\}w\_\{p\(i\)\}\)\\right\\rfloor,ri=fimod\(hp​\(i\)​wp​\(i\)\)r\_\{i\}=f\_\{i\}\\bmod\(h\_\{p\(i\)\}w\_\{p\(i\)\}\),uip=rimodhp​\(i\)u\_\{i\}^\{p\}=r\_\{i\}\\bmod h\_\{p\(i\)\},vip=⌊ri/hp​\(i\)⌋v\_\{i\}^\{p\}=\\left\\lfloor r\_\{i\}/h\_\{p\(i\)\}\\right\\rfloor,uic=mimodhiu\_\{i\}^\{c\}=m\_\{i\}\\bmod h\_\{i\}, andvic=⌊mi/hi⌋v\_\{i\}^\{c\}=\\left\\lfloor m\_\{i\}/h\_\{i\}\\right\\rfloor\. The child position is then recovered asxi=xp​\(i\)\+uip−uicx\_\{i\}=x\_\{p\(i\)\}\+u\_\{i\}^\{p\}\-u\_\{i\}^\{c\},yi=yp​\(i\)\+vip−vicy\_\{i\}=y\_\{p\(i\)\}\+v\_\{i\}^\{p\}\-v\_\{i\}^\{c\}, andzi=zp​\(i\)\+\(1−2​si\)z\_\{i\}=z\_\{p\(i\)\}\+\(1\-2s\_\{i\}\)\. Thus, the attachment tokens provide a reversible local representation of each child brick relative to its parent\.

### A\.2Structure\-aware tree tokenization and detokenization algorithm

Input:A brick assembly

ℬ=\{bi\}i=1N\\mathcal\{B\}=\\\{b\_\{i\}\\\}\_\{i=1\}^\{N\}, where

bi=\(xi,yi,zi,hi,wi\)b\_\{i\}=\(x\_\{i\},y\_\{i\},z\_\{i\},h\_\{i\},w\_\{i\}\)
Output:A token sequence

𝐬\\mathbf\{s\}
1

2Construct a vertical attachment graph

𝒢=\(ℬ,ℰ\)\\mathcal\{G\}=\(\\mathcal\{B\},\\mathcal\{E\}\);

3foreach*pair of bricks\(bi,bj\)\(b\_\{i\},b\_\{j\}\)*do

4if*\|zi−zj\|=1\|z\_\{i\}\-z\_\{j\}\|=1andΩix​y∩Ωjx​y≠∅\\Omega\_\{i\}^\{xy\}\\cap\\Omega\_\{j\}^\{xy\}\\neq\\emptyset*then

5Add edge

\(bi,bj\)\(b\_\{i\},b\_\{j\}\)to

ℰ\\mathcal\{E\};

6

7

8

9Select root brick

br​o​o​t=arg⁡minbi⁡\(zi,yi,xi\)b\_\{root\}=\\arg\\min\_\{b\_\{i\}\}\(z\_\{i\},y\_\{i\},x\_\{i\}\);

10Initialize queue

Q←\[br​o​o​t\]Q\\leftarrow\[b\_\{root\}\];

11Initialize visited set

𝒱←\{br​o​o​t\}\\mathcal\{V\}\\leftarrow\\\{b\_\{root\}\\\};

12Initialize token sequence

𝐬←\[BOS,x0,y0,z0,h0,w0\]\\mathbf\{s\}\\leftarrow\[\\texttt\{BOS\},x\_\{0\},y\_\{0\},z\_\{0\},h\_\{0\},w\_\{0\}\];

13

14while*QQis not empty*do

15Pop current parent brick

bpb\_\{p\}from

QQ;

16Find unvisited neighbors

𝒞​\(bp\)\\mathcal\{C\}\(b\_\{p\}\)of

bpb\_\{p\}in

𝒢\\mathcal\{G\};

17Sort

𝒞​\(bp\)\\mathcal\{C\}\(b\_\{p\}\)by the parent\-side connector index

ff;

18

19foreach*child brickbc∈𝒞​\(bp\)b\_\{c\}\\in\\mathcal\{C\}\(b\_\{p\}\)*do

20Compute parent\-side connector index

ff;

21Compute child\-side anchor index

mm;

22Append

\(f,hc,wc,m\)\(f,h\_\{c\},w\_\{c\},m\)to

𝐬\\mathbf\{s\};

23Add

bcb\_\{c\}to

𝒱\\mathcal\{V\};

24Push

bcb\_\{c\}into

QQ;

25

26

27AppendEOPto

𝐬\\mathbf\{s\};

28

29

30Remove redundant trailingEOPtokens;

31AppendEOSto

𝐬\\mathbf\{s\};

32return

𝐬\\mathbf\{s\};

Algorithm 1Structure\-aware Tree TokenizationInput:A token sequence

𝐬\\mathbf\{s\}
Output:A reconstructed brick assembly

ℬ^\\hat\{\\mathcal\{B\}\}
1

2Construct root brick

b^r​o​o​t=\(x0,y0,z0,h0,w0\)\\hat\{b\}\_\{root\}=\(x\_\{0\},y\_\{0\},z\_\{0\},h\_\{0\},w\_\{0\}\);

3Initialize

ℬ^←\{b^r​o​o​t\}\\hat\{\\mathcal\{B\}\}\\leftarrow\\\{\\hat\{b\}\_\{root\}\\\}and queue

Q←\[b^r​o​o​t\]Q\\leftarrow\[\\hat\{b\}\_\{root\}\];

4

5while*𝐬\\mathbf\{s\}still has remaining tokens*do

6if*QQis empty*then

7break;

8

9

10Pop current parent brick

b^p\\hat\{b\}\_\{p\}from

QQ;

11Initialize child list

𝒞←∅\\mathcal\{C\}\\leftarrow\\emptyset;

12

13while*next token is notEOPand tokens remain*do

14Read child tuple

\(f,hc,wc,m\)\(f,h\_\{c\},w\_\{c\},m\);

15Decode parent\-side connector

\(sf,up,vp\)\(s\_\{f\},u\_\{p\},v\_\{p\}\)from

ff;

16Decode child\-side anchor

\(uc,vc\)\(u\_\{c\},v\_\{c\}\)from

mm;

17

18Recover child horizontal position:

xc←xp\+up−ucx\_\{c\}\\leftarrow x\_\{p\}\+u\_\{p\}\-u\_\{c\},

yc←yp\+vp−vcy\_\{c\}\\leftarrow y\_\{p\}\+v\_\{p\}\-v\_\{c\};

19

20if*sf=0s\_\{f\}=0*then

21Set

zc←zp\+1z\_\{c\}\\leftarrow z\_\{p\}\+1;

22

23else

24Set

zc←zp−1z\_\{c\}\\leftarrow z\_\{p\}\-1;

25

26

27Construct child brick

b^c=\(xc,yc,zc,hc,wc\)\\hat\{b\}\_\{c\}=\(x\_\{c\},y\_\{c\},z\_\{c\},h\_\{c\},w\_\{c\}\);

28Add

b^c\\hat\{b\}\_\{c\}to

ℬ^\\hat\{\\mathcal\{B\}\}and

𝒞\\mathcal\{C\};

29

30

31if*next token isEOP*then

32ConsumeEOP;

33

34

35Push all bricks in

𝒞\\mathcal\{C\}into

QQ;

36

37

38return

ℬ^\\hat\{\\mathcal\{B\}\};

Algorithm 2Structure\-aware Tree Detokenization
### A\.3Structure\-Aware Tree Tokenization Codebook Size and Length Analysis

#### Codebook size\.

We compare the codebook size of the conventional BrickGPT\[[32](https://arxiv.org/html/2605.26182#bib.bib22)\]\-style tokenization and our structure\-aware tree tokenization\. For the BrickGPT\-style baseline, each brick is represented by five tokens, e\.g\.\(h,w,x,y,z\)\(h,w,x,y,z\)\. The coordinate tokens share a discrete resolution of2020, corresponding to coordinate values in the20×20×2020\\times 20\\times 20workspace\. The brick\-size tokens share five possible values,\{1,2,4,6,8\}\\\{1,2,4,6,8\\\}, and three additional special tokens are used forBOS,EOS, andPAD\. Thus, the BrickGPT\-style codebook size is

\|𝒱BrickGPT\|=20\+5\+3=28\|\\mathcal\{V\}\_\{\\mathrm\{BrickGPT\}\}\|=20\+5\+3=28\(22\)
Our structure\-aware tree tokenization uses the same coordinate tokens, brick\-size tokens, and special tokens\. In addition, it introduces oneEOPtoken, parent\-side attachment tokensff, and child\-side anchor tokensmm\. Theff\-token set is determined by the maximum parent footprint area and two vertical attachment directions, while themm\-token set is determined by the maximum child footprint area\. Given our brick library

\{1×1,1×2,1×4,1×6,1×8,2×2,2×4,2×6\}\\\{1\\times 1,1\\times 2,1\\times 4,1\\times 6,1\\times 8,2\\times 2,2\\times 4,2\\times 6\\\}\(23\)
the maximum footprint area is

Amax=2×6=12A\_\{\\max\}=2\\times 6=12\(24\)Therefore, theff\-token set has size2​Amax=242A\_\{\\max\}=24, and themm\-token set has sizeAmax=12A\_\{\\max\}=12\. The total codebook size of our method is

\|𝒱ours\|=20\+5\+3\+1\+24\+12=65\|\\mathcal\{V\}\_\{\\mathrm\{ours\}\}\|=20\+5\+3\+1\+24\+12=65\(25\)

#### Sequence length\.

LetNNbe the number of bricks in a structure\. In the BrickGPT\-style tokenization, each brick is represented by five tokens\. Ignoring padding, the sequence length is

TBrickGPT=5​N\+2T\_\{\\mathrm\{BrickGPT\}\}=5N\+2\(26\)where the two extra tokens correspond toBOSandEOS\.

In our structure\-aware tree tokenization, the root brick is encoded by five absolute tokens\(x0,y0,z0,h0,w0\)\(x\_\{0\},y\_\{0\},z\_\{0\},h\_\{0\},w\_\{0\}\), while each non\-root brick is encoded by four relative tokens\(f,h,w,m\)\(f,h,w,m\)\. In principle, anEOPtoken can be inserted after each parent’s child group\. However, in our experiments, we omit redundantEOPtokens for leaf nodes with no children, since there is no child group to terminate\. LetIIdenote the number of non\-leaf parent nodes that emitEOP\. The sequence length of our tokenization is then

Tours=1\+5\+4​\(N−1\)\+I\+1=4​N\+I\+3T\_\{\\mathrm\{ours\}\}=1\+5\+4\(N\-1\)\+I\+1=4N\+I\+3\(27\)
where the first and last terms correspond toBOSandEOS\. SinceI≤N−1I\\leq N\-1, we have

Tours≤5​N\+2T\_\{\\mathrm\{ours\}\}\\leq 5N\+2\(28\)
which shows that our representation is no longer than the coordinate\-ordered representation, while explicitly encoding parent\-child attachment relations\.

Table 2:Codebook size and sequence length comparison\.The reported average sequence length is computed on the training set\.TokenizationCodebook sizeAvg\. sequence length↓\\downarrowBrickGPT\-style tokenization281050Structure\-aware tree tokenization651044As shown in Table[2](https://arxiv.org/html/2605.26182#A1.T2), our tokenization uses a larger codebook due to the additional attachment tokens, but achieves a slightly shorter average sequence length\.

## Appendix BAdditional Results and Analysis

### B\.1Additional Qualitative Results

Figure[7](https://arxiv.org/html/2605.26182#A2.F7)shows additional generation results on diverse input shapes\. BrickAnything generally preserves the global shape structure while producing physically buildable brick assemblies\.

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/more_results.png)Figure 7:Additional qualitative results\.We provide more examples of BrickAnything on diverse input shapes\. The generated brick structures preserve the overall geometry of the input point clouds while satisfying brick\-level construction constraints\.
### B\.2Failure Cases

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/failureans.png)Figure 8:Failure cases\.We show representative failure cases of BrickAnything\.Figure[8](https://arxiv.org/html/2605.26182#A2.F8)shows representative failure cases\. Failures mainly occur for inputs whose geometry depends on slender components, smooth curved surfaces, or subtle local details, where the limited workspace resolution and restricted brick library make accurate reconstruction difficult\.

### B\.3Image\- and Text\-to\-Brick Results

![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/img2brick.png)Figure 9:Image\-to\-brick results\.Given an input image, we first use Hunyuan3D 2\.5\[[20](https://arxiv.org/html/2605.26182#bib.bib55)\]to reconstruct a 3D mesh, and then convert the mesh into a point cloud with normals as the input condition for BrickAnything\. The generated brick structures preserve the major geometry of the input objects while satisfying brick\-level construction constraints\.![Refer to caption](https://arxiv.org/html/2605.26182v1/figures/text2brick.png)Figure 10:Text\-to\-brick results\.Given an input text prompt, we first use Hunyuan3D 2\.5\[[20](https://arxiv.org/html/2605.26182#bib.bib55)\]to generate a corresponding 3D mesh, and then convert it into a point cloud with normals as the geometric condition for BrickAnything\. The results show that BrickAnything can be extended to text\-conditioned brick generation through an intermediate 3D representation\.Although BrickAnything takes point clouds as input, it can be naturally extended to image\- and text\-conditioned brick generation by using an external 3D generation model as a geometry converter\. Specifically, we feed the input image or text prompt into Hunyuan3D 2\.5\[[20](https://arxiv.org/html/2605.26182#bib.bib55)\]to obtain a corresponding 3D mesh\. The mesh is then converted into a point cloud with normals, which serves as the geometric condition for BrickAnything\. This pipeline allows BrickAnything to generate buildable brick structures from higher\-level modalities while still relying on explicit 3D geometric guidance\.

## Appendix CTraining Details

We pre\-train BrickAnything on 8 NVIDIA A100 80GB GPUs with a batch size of 16 per GPU, giving a global batch size of 128\. The model is trained for 50 epochs, taking approximately one day\. For each mesh, we randomly sample 8192 surface points with normal vectors as the input shape representation\. The autoregressive decoder is trained with fp16 mixed precision using AdamW\. We use a base learning rate of1×10−41\\times 10^\{\-4\}, a final learning rate of6×10−56\\times 10^\{\-5\}, weight decay0\.10\.1, cosine scheduling, first\-epoch warmup from1×10−61\\times 10^\{\-6\}, and gradient clipping with maximum norm 1\.0\. We also apply random scaling, shifting, and rotation augmentations during training\.

For DPO post\-training, we sample 4K training shapes and construct 16K curated preference pairs using the proposed buildability\-aware reward\. The pre\-sampled point clouds corresponding to each mesh are directly loaded during DPO training, without additional mesh resampling\. We fine\-tune the model for 10 epochs on 8 NVIDIA A100 80GB GPUs with a per\-GPU batch size of 8\. The DPO coefficient is set toβ=1\\beta=1, and the auxiliary SFT loss weight is set toλ=1\\lambda=1\. We use a base learning rate of5×10−75\\times 10^\{\-7\}, a final learning rate of1×10−71\\times 10^\{\-7\}, a warmup learning rate of1×10−71\\times 10^\{\-7\}, gradient clipping of 10\.0, and reserve 10% of the preference pairs for validation\.

## Appendix DDetailed Ablation Analysis

We provide a detailed analysis of the ablated variants in Table[1](https://arxiv.org/html/2605.26182#S3.T1)and Figure[5](https://arxiv.org/html/2605.26182#S4.F5)\.

#### Effect of validity\-constrained decoding\.

Validity\-constrained decoding is critical for ensuring that the generated token sequence can be converted into a legal brick structure\. When validity\-constrained decoding, rollback, and DPO are all removed, the model achieves strong geometric scores, reaching0\.12710\.1271CD and0\.6020\.602IoU on the challenging subset\. However, its buildability is substantially weakened, with the stable and valid rates dropping to60\.6%60\.6\\%and68\.6%68\.6\\%, respectively\. A similar trend is observed on the stable subset, where this variant obtains0\.12650\.1265CD and0\.7980\.798IoU, but only reaches63\.4%63\.4\\%stable rate and71\.6%71\.6\\%valid rate\. This indicates that high geometric overlap alone does not guarantee legal or physically realizable brick assemblies\. As shown in Figure[5](https://arxiv.org/html/2605.26182#S4.F5), unconstrained generation may produce invalid token combinations or spatial collisions, which can further lead to detokenization failures\.

#### Effect of rollback\.

Adding validity\-constrained decoding restores the valid rate to100%100\\%on both subsets, confirming its effectiveness in preventing illegal placements\. However, without rollback, the stable rate remains limited:72\.0%72\.0\\%on the challenging subset and91\.0%91\.0\\%on the stable subset\. This shows that local validity constraints can ensure legal brick placements, but they cannot fully guarantee global physical stability\. Rollback complements validity\-constrained decoding by identifying unstable generated structures and resampling structurally related brick placements\. With rollback, the variant without DPO improves the stable rate from72\.0%72\.0\\%to82\.4%82\.4\\%on the challenging subset, and from91\.0%91\.0\\%to100%100\\%on the stable subset\. Thus, validity\-constrained decoding mainly addresses structural legality, while rollback is necessary for correcting stability failures\.

#### Effect of buildability\-aware DPO\.

DPO improves both the BrickGPT\-style baseline and our structure\-aware tree\-tokenized model\. For the BrickGPT\-style baseline, DPO improves the stable rate from74\.0%74\.0\\%to76\.0%76\.0\\%on the challenging subset, increases IoU from0\.5440\.544to0\.5530\.553, and reduces the average number of rollbacks from6\.9916\.991to6\.7506\.750\. On the stable subset, where both variants already achieve100%100\\%stable and valid rates, DPO still improves IoU from0\.7210\.721to0\.7420\.742and reduces rollback average from3\.9163\.916to3\.6203\.620\.

DPO also benefits our tree\-tokenized model\. Compared with the variant without DPO, the full model improves IoU from0\.5730\.573to0\.5860\.586on the challenging subset and from0\.7510\.751to0\.7880\.788on the stable subset\. It also reduces the average number of rollbacks from0\.4440\.444to0\.4220\.422and from0\.2160\.216to0\.1840\.184, respectively\. These results suggest that buildability\-aware preference optimization improves the generation distribution itself, leading to more geometry\-faithful structures and reducing the need for rollback\-based correction\.

#### Effect of structure\-aware tree tokenization\.

Compared with the BrickGPT\-style coordinate\-ordered tokenization, our structure\-aware tree tokenization consistently improves fidelity, stability, and correction efficiency\. Without DPO, our model improves the challenging\-subset IoU from0\.5440\.544to0\.5730\.573, increases the stable rate from74\.0%74\.0\\%to82\.4%82\.4\\%, and reduces rollback average from6\.9916\.991to0\.4440\.444\. With DPO, BrickAnything further improves over the BrickGPT\-style baseline, increasing IoU from0\.5530\.553to0\.5860\.586, improving the stable rate from76\.0%76\.0\\%to83\.4%83\.4\\%, and reducing rollback average from6\.7506\.750to0\.4220\.422\. On the stable subset, BrickAnything also improves IoU from0\.7420\.742to0\.7880\.788and reduces rollback average from3\.6203\.620to0\.1840\.184, while maintaining100%100\\%stable and valid rates\. These results indicate that explicitly modeling local attachment relations yields more coherent brick sequences and substantially reduces the correction burden during inference\.

#### Summary\.

Overall, the ablation results show that the three components address complementary aspects of buildable brick generation\. Validity\-constrained decoding ensures local legality, rollback corrects global stability failures, and buildability\-aware DPO improves the model distribution toward more faithful and stable generations\. In addition, structure\-aware tree tokenization provides a stronger sequence representation than coordinate\-ordered tokenization, leading to higher fidelity, higher stability, and fewer rollback corrections\.

## Appendix ELimitations and Future Work

BrickAnything is currently evaluated in a restricted20×20×2020\\times 20\\times 20workspace with eight standard brick types\. While this setting enables controlled comparison, it also constrains the resolution, part diversity, and geometric complexity of the generated structures\. The failure cases in Fig\.[8](https://arxiv.org/html/2605.26182#A2.F8)further suggest that our current representation is less effective for shapes whose geometry relies on slender components, subtle local variations, or smooth continuous surfaces\. In such cases, the coarse voxelized workspace and limited set of rectangular brick primitives may force the model to produce over\-simplified or structurally distorted approximations\. Future work will extend the framework to larger workspaces, richer brick libraries, and more diverse connection types\.

Our training data is generated by applying Legolization to curated mesh collections, and incorporating human\-designed brick models could improve data diversity and construction realism\. In addition, our buildability\-aware reward is hand\-designed from geometric fidelity and stability metrics\. Learning more adaptive reward models or incorporating real assembly feedback may further improve physical buildability\.