Foundation Models for Automatic CAD Generation

# Foundation Models for Automatic CAD Generation
Source: [https://arxiv.org/html/2607.05573](https://arxiv.org/html/2607.05573)
\\tocauthor

de Curtò et al\.

11institutetext:Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center,
08034 Barcelona, Spain22institutetext:Escuela Técnica Superior de Ingeniería \(ICAI\), Universidad Pontificia Comillas, 28015 Madrid, Spain33institutetext:Department of Artificial Intelligence, Chung\-Ang University, Seoul, Republic of Korea44institutetext:Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L\-4362 Esch\-sur\-Alzette, Luxembourg###### Abstract

Recent advances in Large Language Models \(LLMs\) and Vision\-Language Models \(VLMs\) have opened new pathways for the automatic generation of parametric three\-dimensional designs from natural\-language specifications\. This chapter presents a comprehensive empirical study on the use of modern foundation models for automatic Computer\-Aided Design \(CAD\) generation of mechanical parts, structured around a unified evaluation pipeline and a curated benchmark suite of 97 engineering design problems\. We introduce LLMForge, a multi\-model text\-to\-CAD framework that integrates JSON\-schema validation, analytic feature scoring, mesh synthesis, and a multi\-round iterative refinement loop, studied under two distinct critique regimes\. The first regime, IterTracer, employs a Phong\-shaded ray\-trace renderer coupled with a suite of analytic visual metrics, silhouette IoU, hole visibility, edge clearance, and aspect\-ratio conformance, to provide lightweight, geometry\-aware feedback across successive generation rounds\. The second regime, IterVision, replaces the analytic scorer with a VLM\-based semantic critic \(Qwen2\.5\-VL\-72B\) that evaluates rendered views of each candidate geometry through chain\-of\-thought visual reasoning, enabling richer assessment of spatial coherence and design intent\. Using a benchmark spanning four canonical geometry families, rectangular plates with holes and bolt circles, multi\-feature boxes, flanged cylinders, and L\-brackets, we evaluate seven state\-of\-the\-art foundation models: DeepSeek\-V3\.2, Qwen3\-235B\-A22B, Llama\-3\.3\-70B, Gemma\-3\-27B, GLM\-4\.5, MiniMax\-M2\.1, and INTELLECT\. Under IterTracer, the four highest\-ranked models form a tight performance cluster \(μoverall∈\[0\.885,0\.890\]\\mu\_\{\\text\{overall\}\}\\in\[0\.885,\\,0\.890\]\) with mesh success rates of 98\.97%, demonstrating that compact instruction\-tuned models can attain reliability competitive with substantially larger systems\. The addition of VLM\-based critique in IterVision yields 100% watertight mesh generation on the leading model while surfacing systematic difficulty on rotationally symmetric geometries such as cylinders, where visual and semantic scoring diverge most markedly\. The chapter discusses benchmark design principles, model failure modes, CAD\-oriented prompting strategies, and implications for industrial engineering workflows\. We conclude by identifying future research directions for scalable, automated mechanical design within Global Applied AI pipelines\.

###### keywords:

Large Language Models, Vision\-Language Models, Text\-to\-CAD, Parametric Design, Foundation Models, Engineering Automation

## 1Introduction

The automation of mechanical design has long been a central aspiration of engineering informatics\. Traditional Computer\-Aided Design \(CAD\) workflows demand expert knowledge of constraint\-based parametric modelling, scripting languages, and domain conventions that are largely inaccessible to non\-specialists and difficult to integrate into automated processing chains\. The emergence of large language models capable of generating structured code from natural\-language instructions represents a qualitative shift in this landscape: a practitioner can now describe a mechanical part in plain English,“a 150 mm×\\times100 mm×\\times4 mm plate with a centred bolt circle of six equally\-spaced M5 holes, diameter 60 mm”, and expect a computational agent to translate that intent directly into a valid, manufacturable three\-dimensional geometry\.

This chapter reports on LLMForge \(Language\-Large\-ModelFoundation\-drivenOptimisedRefinementGenerationEngine\), a systematic study of this capability across seven state\-of\-the\-art foundation models examined under two progressively richer evaluation regimes\. The work addresses a practical need that is increasingly common across precision manufacturing and aerospace engineering contexts: large numbers of structural components must be regenerated or adapted from engineering specifications with limited manual intervention\. Automating this process through language\-guided generation and multi\-round iterative critique could substantially reduce design iteration cycles and lower the barrier to rapid prototyping\.

The central challenge is not merely generating syntactically valid code, but generating code that is*semantically correct*with respect to geometric intent\. A CAD script that parses without error may nonetheless produce a geometry with incorrect dimensions, missing features, or topological defects that prevent downstream meshing or simulation\. This motivates the design of LLMForge’s multi\-dimensional evaluation pipeline, which scores generated parts along four independent axes, schema validation, mesh soundness, feature adherence, and visual fidelity, and iterates the generation loop up to four rounds per problem, allowing models to self\-correct in response to structured feedback\.

### Contributions

This chapter makes the following specific contributions\.

1. 1\.IterTracer: an iterative text\-to\-CAD generation pipeline in which each candidate geometry is rendered via a Phong\-shaded ray tracer and scored against a suite of analytic visual metrics \(silhouette IoU, hole visibility, edge clearance, aspect\-ratio conformance\)\. The resulting scalar critique is incorporated into the next\-round prompt, enabling lightweight geometry\-aware feedback without a secondary neural model\. \([Section4](https://arxiv.org/html/2607.05573#S4)\)
2. 2\.IterVision: an extension of the pipeline that replaces the analytic visual scorer with a VLM\-based semantic critic \(Qwen2\.5\-VL\-72B operating in chain\-of\-thought mode\)\. This enables higher\-level assessment of spatial coherence and design intent, and introduces a new*VLM score*dimension into the evaluation record alongside validation, mesh, feature, and visual axes\. \([Section4](https://arxiv.org/html/2607.05573#S4)\)
3. 3\.A benchmark of 97 engineering design problemsspanning four canonical geometry families: rectangular plates with holes and bolt circles, multi\-feature boxes, flanged cylinders, and L\-brackets\. Each problem is paired with a ground\-truth feature specification used to compute analytic feature adherence scores\. \([Section3](https://arxiv.org/html/2607.05573#S3)\)
4. 4\.A comparative empirical evaluation of seven foundation models, DeepSeek\-V3\.2, Qwen3\-235B\-A22B, Llama\-3\.3\-70B, Gemma\-3\-27B, GLM\-4\.5, MiniMax\-M2\.1, and INTELLECT, across both pipelines, producing 679 scored design attempts per system with per\-round convergence trajectories and full mesh\-quality statistics\. \([Section5](https://arxiv.org/html/2607.05573#S5)\)

### Key Findings

Under the IterTracer regime, four models \(DeepSeek\-V3\.2, Qwen3\-235B\-A22B, Llama\-3\.3\-70B, and Gemma\-3\-27B\) achieve overall scores within a remarkably tight band of\[0\.885,0\.890\]\[0\.885,\\,0\.890\], each with a mesh success rate of 98\.97%, suggesting that the text\-to\-CAD task for this class of problems has reached a saturation point for top\-tier instruction\-tuned models under analytic visual critique\. The three remaining models \(GLM\-4\.5, MiniMax\-M2\.1, INTELLECT\) exhibit substantially higher variance \(σ\>0\.28\\sigma\>0\.28\) and lower success rates, reflecting qualitatively different failure modes: schema non\-conformance, degenerate meshes, and persistent feature omission\.

Under the IterVision regime, the addition of VLM\-based critique exerts a more demanding semantic assessment pressure, reflected in a uniform reduction of approximately 0\.04 points in overall score across the top\-four cluster compared to IterTracer\. Despite this, Gemma\-3\-27B achieves a mesh success rate of*100%*\(97/97\), and all 512 analysed meshes are fully watertight and topologically valid solid volumes\. Cylinder geometries emerge as the most challenging category across all models, with visual and VLM scores diverging by up to 0\.15 points relative to plates and L\-brackets, indicating a systematic difficulty with rotationally symmetric features that analytic metrics partially overlook\.

### Scope and Limitations

The benchmark is intentionally bounded to four canonical geometry families to enable systematic comparison; more complex assemblies, freeform surfaces, and multi\-body parts lie outside the current scope\. All experiments are conducted using Nebius AI Studio as the inference backend, with temperature settings held constant across models to ensure comparability\. The VLM scorer in IterVision introduces stochastic evaluation variance that is partially absorbed by the multi\-round protocol but cannot be fully eliminated; we report VLM scores descriptively and treat them as complementary to, rather than replacements for, the analytic metrics\.

### Chapter Organisation

The remainder of this chapter is organised as follows\.[Section2](https://arxiv.org/html/2607.05573#S2)reviews related work on LLM\-assisted design, code generation for CAD, and VLM\-based quality assessment\.[Section3](https://arxiv.org/html/2607.05573#S3)presents the benchmark dataset and evaluation pipeline in detail, covering problem construction, the JSON\-schema CAD representation, scoring components, and the iterative refinement protocol\.[Section4](https://arxiv.org/html/2607.05573#S4)describes the two critique regimes, IterTracer and IterVision, and their respective signal characteristics\.[Section5](https://arxiv.org/html/2607.05573#S5)reports experimental results, including per\-model rankings, part\-type difficulty analysis, round\-level convergence, mesh\-quality statistics, and a comparative analysis between the two systems\.[Section6](https://arxiv.org/html/2607.05573#S6)discusses model failure modes, the effect of model scale and instruction tuning, and the implications for deploying foundation models in industrial CAD pipelines\.[Section7](https://arxiv.org/html/2607.05573#S7)concludes with a research outlook toward scalable, automated mechanical design\.

## 2Related Work

#### Foundation models and code generation\.

The transformer\[[16](https://arxiv.org/html/2607.05573#bib.bib4)\]and the demonstration of few\-shot generalisation in large autoregressive models\[[3](https://arxiv.org/html/2607.05573#bib.bib5)\]underpin all modern code\-generating systems\. Codex\[[5](https://arxiv.org/html/2607.05573#bib.bib9)\]established that models trained on code corpora can complete function bodies from natural\-language docstrings, motivating subsequent work on program synthesis\[[2](https://arxiv.org/html/2607.05573#bib.bib11)\]and multi\-turn generation\[[13](https://arxiv.org/html/2607.05573#bib.bib10)\]\. General\-purpose instruction\-tuned models, including GPT\-4\[[14](https://arxiv.org/html/2607.05573#bib.bib6)\]and the LLaMA family\[[15](https://arxiv.org/html/2607.05573#bib.bib7),[9](https://arxiv.org/html/2607.05573#bib.bib8)\], have since narrowed the gap with specialist code models, making them practical backends for structured\-output tasks such as the JSON\-schema CAD scripts used in LLMForge\. Iterative self\-correction via structured feedback has been shown to improve generation quality without retraining\[[11](https://arxiv.org/html/2607.05573#bib.bib12)\], and LLMs can leverage execution error signals to self\-debug across multiple correction rounds\[[6](https://arxiv.org/html/2607.05573#bib.bib13)\]\. These results directly motivate the multi\-round refinement loop in both IterTracer and IterVision\.

#### Generative models for CAD\.

Parametric CAD generation, producing sequences of modelling operations rather than raw geometry, has attracted growing attention\. DeepCAD\[[19](https://arxiv.org/html/2607.05573#bib.bib18)\]proposed a transformer\-based autoregressive model for CAD construction sequences, and SkexGen\[[20](https://arxiv.org/html/2607.05573#bib.bib19)\]improved generalisability by disentangling topology and geometry into separate codebooks\.\[[12](https://arxiv.org/html/2607.05573#bib.bib20)\]demonstrated that general\-purpose LLMs such as GPT\-4 can assist in parametric design tasks through interactive dialogue, positioning the LLM\-prompting approach as a practical complement to dedicated architectures\. LLMForge extends this line by systematically comparing seven off\-the\-shelf instruction\-tuned models under a unified, reproducible evaluation pipeline across four canonical geometry families\.

#### Vision\-language models and model\-as\-judge evaluation\.

Flamingo\[[1](https://arxiv.org/html/2607.05573#bib.bib14)\], LLaVA\[[10](https://arxiv.org/html/2607.05573#bib.bib15)\], CogVLM\[[18](https://arxiv.org/html/2607.05573#bib.bib16)\], and Qwen2\-VL\[[17](https://arxiv.org/html/2607.05573#bib.bib17)\]have demonstrated that fusing visual and linguistic representations enables rich cross\-modal reasoning\. GPT\-4V\[[14](https://arxiv.org/html/2607.05573#bib.bib6)\]further showed that such models can act as surrogate evaluators of visual artefacts\. Surveys of LLM evaluation\[[4](https://arxiv.org/html/2607.05573#bib.bib21)\]and code\-generation benchmarking\[[21](https://arxiv.org/html/2607.05573#bib.bib22)\]have highlighted the importance of decomposed, multi\-axis scoring over binary pass/fail verdicts, a principle that LLMForge applies to the CAD domain\. The IterVision pipeline instantiates the model\-as\-judge paradigm for parametric CAD by deploying a Qwen2\.5\-VL\-72B critic within the iterative refinement loop, building on earlier LLM applications to engineering reasoning\[[8](https://arxiv.org/html/2607.05573#bib.bib1),[7](https://arxiv.org/html/2607.05573#bib.bib2)\]\.

## 3Benchmark and Evaluation Pipeline

The benchmark consists of 97 engineering design problems encoded as natural\-language strings, drawn from four canonical geometry families: rectangular*plates*with holes and bolt circles, hollow and solid*boxes*,*cylinders*, and*L\-brackets*\. Problems span a range of specification complexity, from a plain“Make a simple 100×\\times80×\\times5 mm rectangular plate”to multi\-feature descriptions combining bolt circles, individual hole offsets, chamfers, and fillets\. Each problem record carries apart\_typelabel and adifficultytag used for stratified analysis; ground\-truth feature specifications \(hole count, nominal dimensions, and surface treatments\) are stored alongside each problem and used to compute the analytic feature adherence score described in[Section3\.1](https://arxiv.org/html/2607.05573#S3.SS1)\.

Foundation models are prompted to emit a strictly structured JSON object rather than free\-form geometry code\. The schema, held constant across all models and both pipelines, is:

```
{
  "part_type"    : "plate"|"box"|"cylinder"|"l_bracket",
  "units"        : "mm",
  "params"       : { ... },
  "holes"        : [{"x":n,"y":n,"diameter":n,"pattern_id":s}],
  "fillet_radius": number,
  "chamfer"      : number,
  "transform"    : {"rx":0,"ry":0,"rz":0,"tx":0,"ty":0,"tz":0}
}
```

Theparamsblock is keyed onpart\_type:platerequireswidth,height, andthickness;boxaddsdepthand an optionalthickness\(null for a solid box\);cylinderrequiresdiameterandheight;l\_bracketrequires leg dimensionsa,b,thickness, andleg\_width\. All quantities are in millimetres\. The initial system prompt enforces metric\-to\-clearance mappings for threaded holes \(e\.g\. M3→\\to3\.4 mm, M4→\\to4\.5 mm, M5→\\to5\.5 mm, M6→\\to6\.6 mm, M8→\\to9\.0 mm\) and requires hole coordinates to be expressed relative to the part centre\. Models must return the JSON object only, with no markdown fences or surrounding text; a regex\-guarded parser strips residual formatting before further processing\.

A deterministic Python geometry engine converts each valid JSON spec into a watertight triangulated mesh using Trimesh and Shapely\. For plates, a 2\-D rectangular Shapely polygon is constructed at the nominal dimensions and each hole is subtracted as a 96\-segment circular buffer at the declared centre and clearance diameter; the resulting polygon \(resolved through abuffer\(0\)validity repair if needed\) is then extruded to the declared thickness\. For boxes, a solid box primitive is created with Trimesh’screation\.box; if a wall thickness is specified, an inner box is subtracted via Boolean difference\. Cylinders are tessellated with 96 circumferential sections\. L\-brackets are assembled from two box primitives: a horizontal base leg and a vertical leg obtained by rotating the second box90∘90^\{\\circ\}about theyy\-axis, then concatenating both into a single mesh object\. After construction, degenerate faces, unreferenced vertices, and duplicate vertices are removed\.

### 3\.1Scoring Axes

Each generated part is evaluated along four independent axes\.

Validation score\(sval∈\[0,1\]s\_\{\\text\{val\}\}\\in\[0,1\]\) measures JSON schema conformance: presence of all required keys, correctpart\_typevalue, numeric types for all parameters, and absence of structurally invalid entries\.

Mesh score\(smesh∈\{0,1\}s\_\{\\text\{mesh\}\}\\in\\\{0,1\\\}\) is a binary indicator that equals 1 if and only if the geometry engine produces a non\-empty mesh without raising an exception, and 0 otherwise\.

Feature score\(sfeat∈\[0,1\]s\_\{\\text\{feat\}\}\\in\[0,1\]\) compares the generated specification against the ground\-truth feature record: hole count, principal dimensions, and declared surface treatments \(chamfer, fillet\) are checked analytically, and partial credit is awarded proportionally\.

Visual score\(svis∈\[0,1\]s\_\{\\text\{vis\}\}\\in\[0,1\]\) is a composite of five sub\-signals derived from rendered bitmaps of the candidate mesh: silhouette IoU \(fill ratio of part pixels within the tight bounding box of the top\-view projection\), hole visibility \(fraction of declared holes whose projections are visible from the top\-view render\), edge clearance \(minimum distance from any hole projection to the plate boundary\), aspect\-ratio conformance, and a cross\-sectional consistency score\. The sub\-signal weights are: silhouette IoU 0\.30, hole visibility 0\.30, edge clearance 0\.20, aspect ratio 0\.10, section score 0\.10\.

The composite overall score differs between the two pipelines\. Under IterTracer:

soverallA=0\.25​sval\+0\.15​smesh\+0\.20​sfeat\+0\.40​svis\.s\_\{\\text\{overall\}\}^\{\\text\{A\}\}=0\.25\\,s\_\{\\text\{val\}\}\+0\.15\\,s\_\{\\text\{mesh\}\}\+0\.20\\,s\_\{\\text\{feat\}\}\+0\.40\\,s\_\{\\text\{vis\}\}\.\(1\)Under IterVision, a fifth axis, the VLM semantic matchsvlms\_\{\\text\{vlm\}\}returned by the Qwen2\.5\-VL\-72B critic, is incorporated:

soverallB=0\.20​sval\+0\.10​smesh\+0\.20​sfeat\+0\.30​svis\+0\.20​svlm\.s\_\{\\text\{overall\}\}^\{\\text\{B\}\}=0\.20\\,s\_\{\\text\{val\}\}\+0\.10\\,s\_\{\\text\{mesh\}\}\+0\.20\\,s\_\{\\text\{feat\}\}\+0\.30\\,s\_\{\\text\{vis\}\}\+0\.20\\,s\_\{\\text\{vlm\}\}\.\(2\)The reweighting reduces the load on the analytic visual axis from 0\.40 to 0\.30 and redistributes 0\.20 to the VLM axis\.

### 3\.2Iterative Refinement Protocol

Both pipelines follow the same iterative protocol, instantiated asIterativeCADEvaluatorin the accompanying notebooks\. Each problem is processed for up toR=3R=3refinement rounds indexedr∈\{0,1,2,3\}r\\in\\\{0,1,2,3\\\}\. Round 0 uses the initial system prompt \(SYSTEM\_INITIAL\) to generate a first\-pass JSON spec from the plain\-text description\. Roundsr≥1r\\geq 1use a refinement system prompt \(SYSTEM\_REFINE\) that injects three structured feedback blocks into the user message:

1. 1\.Structural feedback: schema errors, missing or incorrect features identified by the analytic scorer, and edge clearance violations\.
2. 2\.Visual feedback: the five sub\-signals from the ray\-trace critic \(silhouette IoU, hole visibility, edge clearance visual score, aspect ratio, composite\)\.
3. 3\.VLM semantic feedback\(IterVision only, roundsr≤2r\\leq 2\): the structured JSON response from Qwen2\.5\-VL\-72B, includingmissing\_features,incorrect\_features,geometry\_issues, and actionablesuggestions\.

The LLM backend is queried via the Nebius AI Studio API \(OpenAI\-compatible endpoint\) at temperature 0\.15 and a maximum of 2 048 output tokens, with up to three retries per call with exponential back\-off\. The best\-scoring spec across all rounds is retained as the final output \(*best\-of\-NN*selection\); an early\-exit criterion halts refinement if the composite score reaches 0\.92 or above\. The refinement prompt explicitly instructs the model to preserve all holes, enforce a minimum edge clearance of1\.5×1\.5\\timesthe maximum hole radius, correct schema errors, and add a fillet radius of 5–10% of the shorter principal dimension if absent\.

### 3\.3Models and Experimental Conditions

Seven foundation models available on Nebius AI Studio are evaluated under both pipelines on the same 97\-problem benchmark: Llama\-3\.3\-70B\-Instruct, DeepSeek\-V3\.2, INTELLECT\-3, Qwen3\-235B\-A22B\-Instruct, Gemma\-3\-27B\-IT, GLM\-4\.5, and MiniMax\-M2\.1\. All models are queried with the same temperature \(T=0\.15T=0\.15\), token budget \(2 048\), and refinement protocol\. A cooldown of 5 s is inserted between consecutive model runs to avoid rate\-limit interference\. Per\-problem STL files of the best\-round mesh are exported alongside per\-model JSON result files recording round\-level trajectories for all scoring axes\.

![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/mesh_gallery2.png)Figure 1:Example output meshes generated by DeepSeek\-V3\.2 under IterVision \(best\-round STL exports\), rendered from three orthographic viewpoints: isometric, top \(XY\), and front \(XZ\)\. From top to bottom: a coaxial\-connector mounting plate \(90×\\times70×\\times4 mm, 5 clearance holes\) and a microwave\-cavity end plate \(150×\\times150×\\times10 mm, central aperture with bolt circle\)\. All meshes are watertight closed volumes; hole edges and Phong shading confirm correct feature geometry\.

## 4Critique Regimes: IterTracer and IterVision

Under IterTracer, the visual feedback at each refinement round is produced entirely by deterministic, differentiable geometric analysis of the rendered bitmap, no secondary neural model is involved\. After the geometry engine constructs the candidate mesh, two orthographic views \(top and isometric\) are rendered at256×256256\\times 256pixels using a Matplotlib 3\-D backend with Phong\-like face colouring\. The top\-view silhouette is extracted by thresholding non\-white pixels and used to compute: \(i\) the*silhouette IoU*, defined as the fraction of part pixels within the tight axis\-aligned bounding box of the projection, which serves as a proxy for mesh coherence and dimensional completeness; \(ii\)*hole visibility*, the fraction of declared holes whose circular projections remain detectable in the top\-view bitmap; \(iii\)*edge clearance*, the minimum normalised distance from any visible hole projection to the plate boundary; and \(iv\)*aspect\-ratio conformance*, the relative deviation between the rendered width\-to\-height ratio and the declared parameter ratio\. A fifth sub\-signal,*section score*, aggregates cross\-sectional consistency checks\. All five sub\-signals are linearly combined into a single visual composite \(weights listed in[Section3\.1](https://arxiv.org/html/2607.05573#S3.SS1)\), which feeds directly into the composite scoresoverallAs\_\{\\text\{overall\}\}^\{\\text\{A\}\}\([Equation1](https://arxiv.org/html/2607.05573#S3.E1)\) and is reported verbatim to the model as numerical feedback in the next\-round refinement prompt\. The principal advantage of this regime is speed and determinism: critique latency is sub\-second, introduces no stochastic variance, and does not require additional API calls\.

IterVision augments the analytic critic with a Qwen2\.5\-VL\-72B semantic inspector deployed on Nebius AI Studio\. The renderer is upgraded to a full Phong\-shading pipeline: per\-face normals are computed analytically, a Blinn\-Phong model with ambientka=0\.25k\_\{a\}=0\.25, diffusekd=0\.65k\_\{d\}=0\.65, and specularks=0\.10k\_\{s\}=0\.10\(shininess 8\) is applied, and the light direction is co\-aligned with the camera\. Three views are rendered at384×384384\\times 384pixels \(top, isometric, front\) and base64\-encoded before being sent to the VLM\. The VLM receives a structured multimodal prompt containing: the original natural\-language specification, the three rendered views, the current JSON spec summary, and the automated visual metrics from the analytic sub\-pipeline\. It is instructed to respond with a single JSON object carryingsemantic\_match\(∈\[0,1\]\\in\[0,1\]\),missing\_features,incorrect\_features,geometry\_issues,suggestions, andconfidence\. Thesemantic\_matchvalue becomessvlms\_\{\\text\{vlm\}\}and enters the composite score \([Equation2](https://arxiv.org/html/2607.05573#S3.E2)\); the textual fields are concatenated into a third feedback block in the refinement prompt, explicitly ranked above structural and analytic visual feedback in the system\-prompt priority order\. VLM critique is active only on refinement roundsr≤2r\\leq 2to limit token expenditure; the total weight of the analytic visual axis is reduced from 0\.40 to 0\.30 to make room for the 0\.20 VLM axis\. A separatevlm\_clientinstance is initialised at VLM temperature 0\.05 to maximise determinism in the scoring output\.

## 5Experimental Results

[Table1](https://arxiv.org/html/2607.05573#S5.T1)reports per\-model mean scores across all 97 problems for both pipelines\. Under IterTracer, four models, DeepSeek\-V3\.2, Qwen3\-235B\-A22B, Gemma\-3\-27B, and Llama\-3\.3\-70B, form a tight cluster in\[0\.885,0\.890\]\[0\.885,\\,0\.890\]with standard deviationσ≤0\.074\\sigma\\leq 0\.074and uniform mesh success rates of 98\.97%\. The average number of refinement rounds consumed by this cluster ranges from 3\.58 to 3\.65, indicating that the protocol rarely triggers an early exit, with the best score typically arising from a later round rather than round 0\. GLM\-4\.5 occupies a distinct second tier at 0\.678 \(σ=0\.305\\sigma=0\.305\), while MiniMax\-M2\.1 and INTELLECT trail at 0\.575 and 0\.411 respectively, with high variance and sub\-50% mesh success rates\.

Under IterVision, the top\-four cluster shifts downward by approximately 0\.040 points uniformly, with overall scores in\[0\.842,0\.850\]\[0\.842,\\,0\.850\], reflecting the more demanding semantic assessment pressure introduced by the VLM axis\. Gemma\-3\-27B is the sole model to achieve a mesh success rate of100%\(97/97\), and all 512 STL files produced by successful evaluations are fully watertight and topologically valid closed volumes\.

Representative output geometries generated by DeepSeek\-V3\.2 are shown in[Figure1](https://arxiv.org/html/2607.05573#S3.F1), illustrating the range of hole patterns, slot features, and dimensional proportions faithfully captured by the pipeline across four distinct plate specifications\.

The VLM axis rankings differ noticeably from the analytic\-axis rankings: DeepSeek\-V3\.2 leads onsvlms\_\{\\text\{vlm\}\}\(μ=0\.625\\mu=0\.625\), while Gemma\-3\-27B scores lowest among the top four \(μ=0\.511\\mu=0\.511\), suggesting that higher mesh reliability does not automatically translate to stronger semantic correspondence as judged by the VLM inspector\.

Table 1:Model performance summary\. IterTracer uses four scoring axes; IterVision adds the VLM semantic axis\.μ\\mu= mean,σ\\sigma= std, SR = mesh success rate,r¯\\bar\{r\}= average rounds\.[Figure2](https://arxiv.org/html/2607.05573#S5.F2)plots all five scoring axes per model under IterVision\. The validation and mesh axes are near\-ceiling \(\>0\.97\>0\.97\) for the top\-four cluster, confirming that schema conformance and geometric validity are essentially saturated at this difficulty level\. The feature axis is more discriminating \(0\.8760\.876–0\.8880\.888for top four;0\.6580\.658–0\.7940\.794for the remaining three\), and the visual axis produces the largest absolute gap between tiers\. Pooled across models, the Pearson correlations withsoveralls\_\{\\text\{overall\}\}are:svis=0\.959s\_\{\\text\{vis\}\}=0\.959,smesh=0\.951s\_\{\\text\{mesh\}\}=0\.951,sval=0\.914s\_\{\\text\{val\}\}=0\.914,svlm=0\.776s\_\{\\text\{vlm\}\}=0\.776,sfeat=0\.722s\_\{\\text\{feat\}\}=0\.722\. The notably lower correlation ofsvlms\_\{\\text\{vlm\}\}\(r=0\.776r=0\.776\) compared tosviss\_\{\\text\{vis\}\}\(r=0\.959r=0\.959\) confirms that the VLM critic captures a partially orthogonal signal: it penalises semantic mismatches that pass analytic visual filters, at the cost of introducing stochastic scoring variance \([Figure4](https://arxiv.org/html/2607.05573#S5.F4)\)\.

Pairwise Bonferroni\-corrected Mann\-Whitney tests onsoveralls\_\{\\text\{overall\}\}show no statistically significant differences within the top\-four cluster \(all adjustedp=1\.0p=1\.0\), while all comparisons between any top\-four model and the lower three are significant \(padj≈0\.0p\_\{\\text\{adj\}\}\\approx 0\.0\)\. GLM\-4\.5 vs\. MiniMax\-M2\.1 are not significantly different \(padj=1\.0p\_\{\\text\{adj\}\}=1\.0\), but both differ significantly from INTELLECT \(padj=0\.0026p\_\{\\text\{adj\}\}=0\.0026and0\.00020\.0002respectively\)\.

![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/tifon_fig_final_comparison_o.png)\(a\)IterTracer \(4 axes\)\.
![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/tifon40_fig_final_comparison_o.png)\(b\)IterVision \(5 axes, VLM added\)\.

Figure 2:Mean score per axis per model under both critique regimes \(97 problems each\)\. Comparing \(a\) and \(b\) shows that the VLM axis introduces new headroom and reorders models on the semantic dimension\.![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/tifon_fig_trajectory.png)\(a\)IterTracer\.
![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/tifon40_fig_trajectory.png)\(b\)IterVision\.

Figure 3:Per\-round score trajectories \(±\\pmSEM\) under both critique regimes\. The qualitative reversal between \(a\) and \(b\) for top\-tier models demonstrates that VLM feedback sustains productive refinement beyond round 1 whereas analytic feedback saturates\.[Figure3](https://arxiv.org/html/2607.05573#S5.F3)shows per\-round mean overall score and VLM semantic\-match trajectories under IterVision\. For the top\-four cluster, the overall score exhibits a shallow dip at round 1 followed by monotonic recovery; 53–55% of problems for DeepSeek\-V3\.2 and Qwen3\-235B\-A22B achieve their best score at round 2, with a further 31–38% at round 3, confirming genuine multi\-round improvement under VLM\-augmented critique\.

The convergence regime under IterTracer is qualitatively different\. Top\-four models peak at round 1 for 62–83% of problems \(Qwen: 83%, Gemma: 71%, Llama: 65%, DeepSeek: 62%\) and their per\-round trajectories decline monotonically thereafter; medianΔ\\Deltafor the top cluster is 0\.025 under IterTracer versus∼\\sim0\.07 under IterVision\. The analytic critic provides a strong one\-shot corrective signal, schema errors and clearance violations repaired in round 1, but its numerical feedback is insufficiently expressive to sustain further refinement, whereas the VLM’s natural\-language critique maintains productive improvement across later rounds\. The correlation structure also shifts: under IterTracer, bothsmeshs\_\{\\text\{mesh\}\}andsviss\_\{\\text\{vis\}\}reachr=0\.978r=0\.978with overall \(versus 0\.951 and 0\.959 under IterVision; see[Figure4](https://arxiv.org/html/2607.05573#S5.F4)\), indicating that without a semantic axis the two analytic measures collapse to essentially co\-linear proxies\.

For weaker models, the refinement benefit is larger in absolute terms under both pipelines: medianΔ\\Deltareaches 0\.19–0\.21 for GLM\-4\.5 and MiniMax\-M2\.1 under IterVision, and 0\.16–0\.18 under IterTracer, confirming that the feedback loop recovers a substantially larger fraction of initially failed attempts for lower\-tier models\. The VLM trajectory \(right panel,[Figure3](https://arxiv.org/html/2607.05573#S5.F3)\) reveals a striking anomaly: INTELLECT’ssvlms\_\{\\text\{vlm\}\}rises steeply from 0\.60 to 0\.82 across two VLM\-active rounds, substantially outpacing all other models on this axis despite ranking last onsoveralls\_\{\\text\{overall\}\}\. This divergence, also visible in the raw VLM score distributions in[Figure2](https://arxiv.org/html/2607.05573#S5.F2), suggests that INTELLECT generates geometries whose rendered appearance is rated favourably by the VLM critic despite failing mesh validation, a form of*visual–geometric decoupling*that highlights the complementary nature of the two critique modalities\.

![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/tifon_fig_correlation.png)\(a\)IterTracer \(4 axes\)\.
![Refer to caption](https://arxiv.org/html/2607.05573v1/drcoyz/tifon40_fig_correlation.png)\(b\)IterVision \(5 axes\)\.

Figure 4:Pearson correlation matrices \(pooled across all models and problems\) for the two critique regimes\. Under IterTracer, mesh and visual scores are near\-redundant; the VLM axis introduced in IterVision breaks this co\-linearity and captures residual semantic variance invisible to analytic metrics\.
## 6Discussion

#### Critique modality determines optimal refinement depth\.

A cross\-pipeline comparison reveals that the type of feedback signal, not simply its presence, determines when iterative refinement stops being beneficial\. Under IterTracer, top\-tier models saturate at round 1 and regress on subsequent rounds; under IterVision, the same models continue improving through rounds 2–3\. This suggests a general principle: analytic feedback corrects discrete structural violations in a single pass, while semantic visual feedback provides a richer, continuous quality gradient that supports deeper search\. For production deployment, this implies that a two\-stage strategy, analytic critique for rapid constraint satisfaction followed by selective VLM invocation for high\-value or ambiguous parts, could substantially reduce per\-problem token expenditure without sacrificing quality on the structurally simple majority\.

#### Saturation of top\-tier models under analytic critique\.

The near\-identical performance of DeepSeek\-V3\.2, Qwen3\-235B\-A22B, Llama\-3\.3\-70B, and Gemma\-3\-27B under IterTracer \(Δoverall<0\.005\\Delta\_\{\\text\{overall\}\}<0\.005across the cluster,padj=1\.0p\_\{\\text\{adj\}\}=1\.0\) suggests that the four\-axis analytic benchmark is essentially saturated for state\-of\-the\-art instruction\-tuned models on this class of canonical geometries\. The addition of the VLM semantic axis in IterVision restores approximately 0\.04 points of headroom and reorders the cluster \(Gemma\-3\-27B drops to rank 4 on VLM despite perfect mesh success\), demonstrating that semantic visual inspection surfaces genuine residual variation that analytic metrics cannot capture\.

#### Failure mode taxonomy\.

The lower tier exhibits two qualitatively distinct failure modes\. GLM\-4\.5 produces valid meshes in∼\\sim55% of cases but with high variance \(σ=0\.305\\sigma=0\.305/0\.2760\.276\), suggesting intermittent schema non\-conformance or dimensional errors rather than systematic inability\. MiniMax\-M2\.1 and INTELLECT suffer more fundamental geometric failures: sub\-50% mesh success rates indicate that the geometry engine frequently raises exceptions on their output, pointing to malformed JSON parameters \(e\.g\. zero\-size dimensions, inverted hole coordinates\) that schema validation cannot prevent\. INTELLECT’s anomalously high VLM scores despite near\-floor geometric success expose a discrepancy between what the VLM perceives as plausible geometry in a Phong\-rendered image and what the analytic pipeline enforces as valid topology\.

#### Geometry\-type difficulty and the cylinder gap\.

Across both pipelines, cylinder geometries score 0\.04–0\.07 points below plates and L\-brackets for the top\-four models\. We attribute this to two compounding factors: \(i\) the absence of planar hole features removes the strongest visual discriminators \(hole visibility, edge clearance\) from the cylinder score, lowering the effective information content of the visual critique; and \(ii\) the VLM critic tends to rate cylindrical renders more conservatively, possibly due to under\-representation of rotating\-symmetric industrial components in its pretraining data\. This geometry\-type gap suggests that future benchmark versions should include flanged and threaded cylinder variants with richer feature sets to provide cleaner diagnostic signals\.

#### Industrial applicability\.

The results indicate that top\-tier instruction\-tuned foundation models can generate dimensionally and topologically valid parametric parts from natural\-language descriptions with near\-98% mesh success rates within three refinement iterations, at temperature 0\.15 without fine\-tuning or retrieval augmentation\. The structured JSON representation and the multi\-round critique protocol are both amenable to integration into engineering PDM workflows as a first\-pass geometry generation module, with human verification reserved for VLM\-flagged semantic mismatches\.

## 7Conclusion

This chapter presented LLMForge, a multi\-model text\-to\-CAD evaluation framework studied under two critique regimes: IterTracer, which employs analytic ray\-trace visual metrics, and IterVision, which augments the analytic critic with a Qwen2\.5\-VL\-72B semantic inspector within the multi\-round refinement loop\. Across a benchmark of 97 engineering design problems, four state\-of\-the\-art instruction\-tuned models achieve near\-identical performance under analytic critique \(μoverall≈0\.887\\mu\_\{\\text\{overall\}\}\\approx 0\.887, mesh success 98\.97%\), with Gemma\-3\-27B reaching 100% watertight mesh success under VLM\-augmented critique\. The VLM axis surfaces a partially orthogonal quality signal \(r=0\.776r=0\.776with overall score vs\.r=0\.959r=0\.959for analytic visual\), restores meaningful score headroom saturated under analytic\-only evaluation, and exposes a visual–geometric decoupling phenomenon in lower\-tier models\. Iterative refinement provides consistent incremental benefit: 53–55% of top\-model problems attain their best score at round 2 or later, and weaker models recover medianΔ≈0\.19\\Delta\\approx 0\.19through the feedback loop\.

Future work will extend the benchmark to freeform surfaces, multi\-body assemblies, and manufacturing\-specific constraints \(GD&T tolerances, material specifications\), and explore fine\-tuning foundation models directly on the structured feedback signal to narrow the gap between top\-tier and lower\-tier models\. Integration with physics simulation backends, replacing rendered image critique with FEA\-based structural validation, represents a natural pathway toward fully automated, simulation\-in\-the\-loop mechanical design generation\.

## Acknowledgments

This research was supported by the LUXEMBOURG Institute of Science and Technology through the projects ‘ADIALab\-MAST’ and ‘LLMs4EU’ \(Grant Agreement No 101198470\) and the BARCELONA Supercomputing Center through the project ‘TIFON’ \(File number MIG\-20232039\)\. Victoria Guillén would also like to thank Universidad Pontificia Comillas for the opportunity to participate in the international exchange program with Chung\-Ang University, Seoul, Republic of Korea\.

## Code Availability

The implementation of the LLMForge framework is publicly available at:https://github\.com/drdecurto/LLMforge\. The repository includes benchmark datasets, the geometry engine, both critique\-regime notebooks \(IterTracer and IterVision\), evaluation pipelines, per\-model result files, and reproduction instructions\.

## References

- \[1\]J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.Advances in Neural Information Processing Systems35,pp\. 23716–23736\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[2\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]T\. Brown, B\. Mann, N\. Ryder,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]Y\. Chang, X\. Wang, J\. Wang, Y\. Wu, L\. Yang, K\. Zhu, H\. Chen, X\. Yi, C\. Wang, Y\. Wang,et al\.\(2024\)A survey on evaluation of large language models\.ACM Transactions on Intelligent Systems and Technology15\(3\),pp\. 1–45\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[5\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2024\)Teaching large language models to self\-debug\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]J\. de Curtò, I\. de Zarzà, and C\. T\. Calafate\(2025\)Integrating polyglot persistence with large language models for scalable social network applications\.Procedia Computer Science270,pp\. 733–743\.Note:29th International Conference on Knowledge\-Based and Intelligent Information & Engineering Systems \(KES 2025\)External Links:ISSN 1877\-0509Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[8\]J\. de Curtò, I\. de Zarzà, and C\. T\. Calafate\(2025\)LLM multi\-agent decision optimization\.InAgents and Multi\-agent Systems: Technologies and Applications 2024,G\. Jezic, Y\.\-H\. Chen\-Burger, M\. Kušek, R\. Šperka, R\. J\. Howlett, and L\. C\. Jain \(Eds\.\),Singapore,pp\. 3–15\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[9\]A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[11\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.Advances in Neural Information Processing Systems36\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]L\. Makatura, M\. Foshey, B\. Wang, F\. Hähnlein, P\. Ma, B\. Deng, M\. Tjandrasuwita, A\. Spielberg, J\. B\. Owens, P\. Y\. Chen,et al\.\(2023\)How can large language models help humans in design and manufacturing?\.arXiv preprint arXiv:2307\.14377\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]E\. Nijkamp, B\. Pang, H\. Hayashi, L\. Tu, H\. Wang, Y\. Zhou, S\. Savarese, and C\. Xiong\(2023\)CodeGen: an open large language model for code with multi\-turn program synthesis\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]OpenAI\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 5998–6008\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-VL: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]W\. Wang, Q\. Lv, W\. Yu, W\. Hong, J\. Qi, Y\. Wang, J\. Ji, Z\. Yang, L\. Zhao,et al\.\(2024\)CogVLM: visual expert for pretrained language models\.Advances in Neural Information Processing Systems37,pp\. 121475–121499\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.
- \[19\]R\. Wu, C\. Xiao, and C\. Zheng\(2021\)DeepCAD: a deep generative network for computer\-aided design models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 6772–6782\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]X\. Xu, K\. D\.D\. Willis, J\. G\. Lambourne, C\. Cheng, P\. K\. Jayaraman, and Y\. Furukawa\(2022\)SkexGen: autoregressive generation of cad construction sequences with disentangled codebooks\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 24698–24724\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]T\. Y\. Zhuo, M\. C\. Vu, J\. Chim, H\. Hu, W\. Liu,et al\.\(2024\)BigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.arXiv preprint arXiv:2406\.15877\.Cited by:[§2](https://arxiv.org/html/2607.05573#S2.SS0.SSS0.Px3.p1.1)\.