Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints

# Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints
Source: [https://arxiv.org/html/2608.17843](https://arxiv.org/html/2608.17843)
###### Abstract

Large language models \(LLMs\) have demonstrated strong performance on structured reasoning tasks, but what they encode and whether it informs model behavior remain unclear\. We investigate this question through geometric reasoning, using parametric CAD constraints as a controlled testbed for separating local pairwise relations from sketch\-level constraint status\. By probing the hidden states of six frozen decoder\-only LLMs, we examine four properties: linear decodability, forced\-choice generation, activation\-level influence, and behavioral steerability\. Pretraining substantially improves the decoding of local geometric relations, and this advantage persists after accounting for positional cues with shuffled\-order controls\. In contrast, sketch\-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining, indicating that much of its probe performance is available without learned weights\. Further analyses show that decodable information is not always actionable\. Generation often fails to express this information, and on the two intervention\-tested backbones, activation\-restoration effects at the patched entity position vanish while decodability persists across depth\. Mean\-difference steering also does not reliably control outputs\. These results show that decodability, generation, activation\-level influence, and steerability can diverge in the tested setting\. The audit provides a controlled way to distinguish failures to encode geometric structure from failures to express or control encoded information\.

## 1Introduction

Large language models are increasingly used to translate natural language instructions into structured outputs such as programs, plans, and geometric designs\([37](https://arxiv.org/html/2608.17843#bib.bib25);[30](https://arxiv.org/html/2608.17843#bib.bib26);[38](https://arxiv.org/html/2608.17843#bib.bib27)\)\. Trustworthy deployment in scientific and industrial settings requires structured outputs to satisfy interacting domain constraints, including physical principles, geometric relations, and safety requirements, while remaining globally consistent\. Among these applications, parametric CAD provides a particularly useful case because it is widely used in engineering and represents domain constraints in an explicit, verifiable form\([27](https://arxiv.org/html/2608.17843#bib.bib1)\)\. In parametric CAD, local constraints define relations between geometric elements, while their combined effect determines whether a sketch is under\-constrained, well\-constrained, or over\-constrained\([31](https://arxiv.org/html/2608.17843#bib.bib28)\)\.

Recent work increasingly uses LLMs to generate CAD designs from natural language instructions\([15](https://arxiv.org/html/2608.17843#bib.bib5);[18](https://arxiv.org/html/2608.17843#bib.bib23)\)\. Newer systems extend one\-shot generation with execution or solver feedback to iteratively detect and repair errors\([4](https://arxiv.org/html/2608.17843#bib.bib29);[12](https://arxiv.org/html/2608.17843#bib.bib30);[20](https://arxiv.org/html/2608.17843#bib.bib31)\)\. However, output\-level evaluation alone does not reveal whether successful generation and repair rely on internal constraint representations\. Clarifying this relationship is necessary to explain why local competence may coexist with global failure\. We therefore ask what geometric constraint information frozen general\-purpose LLMs encode and how its decodability relates to generation, activation\-level influence, and behavioral control\.

To answer this question, we develop a four\-part audit of linear decodability, forced\-choice generation, activation\-level influence, and steering using SketchGraphs\([27](https://arxiv.org/html/2608.17843#bib.bib1)\), with a cross\-dataset P1 check on Fusion 360 Gallery\([36](https://arxiv.org/html/2608.17843#bib.bib42)\)\. Our evaluation covers six frozen decoder\-only LLMs from the Qwen2\.5, Mistral, and Llama\-3\.1 families\([25](https://arxiv.org/html/2608.17843#bib.bib32);[14](https://arxiv.org/html/2608.17843#bib.bib16);[7](https://arxiv.org/html/2608.17843#bib.bib6)\)\. The auditing framework includes three matched evaluation tasks: local decoding of pairwise constraints \(P1\), global decoding of degrees\-of\-freedom status \(P2\), and forced\-choice generation using the same pairwise labels as P1 \(P3\)\. We further use activation patching to test whether restoring activations at the probed entity position affects predictions, and representation steering to test whether modifying those activations can systematically control outputs\. To target latent geometric information rather than explicit label cues, we serialize only the geometry and exclude all constraint annotations from the model input\.

Our results reveal a systematic gap between encoded and actionable geometric constraint information\. Local pairwise relations \(P1\) contain learned information that can be linearly decoded even after accounting for random initialization, input structure, and entity order\. Global constraint status \(P2\), however, benefits little from pretraining\. This local\-to\-global asymmetry is captured by a dissociation index that is positive across all six models\. More importantly, strong P1 decodability does not translate into reliable P3 generation, with failure severity varying sharply across model backbones\. Activation patching reveals that prediction sensitivity to restoration at the patched entity position is concentrated in early layers and vanishes while decodability persists\. Mean\-difference steering does not reliably control predictions\. These mismatches show why failures on structured tasks should be audited at multiple levels rather than attributed to missing knowledge alone\. Our main contribution is a controlled, cross\-model framework for auditing geometric constraint representations in frozen LLMs, distinguishing failures to encode information from failures to express, use, or control it\.

![Refer to caption](https://arxiv.org/html/2608.17843v1/figures/CAD-overview.png)Figure 1:Overview of the four\-part audit\. Geometry\-only sketches are serialized and passed through a frozen LLM, whose representations are evaluated for linear decodability, forced\-choice generation, sensitivity to activation restoration, and steering\-based control\.
## 2Related Work

#### Geometric reasoning and CAD generation\.

Geometric constraint reasoning underlies engineering tasks such as 3D object placement, robotic assembly planning, and manufacturing workflows\([13](https://arxiv.org/html/2608.17843#bib.bib17);[16](https://arxiv.org/html/2608.17843#bib.bib19);[5](https://arxiv.org/html/2608.17843#bib.bib18)\)\. Parametric CAD makes this local\-to\-global structure explicit: pairwise geometric constraints define local relations, while remaining degrees of freedom characterize sketch\-level constraint status\.

Prior work captures this structure through complementary representations\. SketchGraphs models primitives and explicit pairwise constraints\([27](https://arxiv.org/html/2608.17843#bib.bib1)\), whereas Fusion 360 Gallery, DeepCAD, and CADParser represent designs through construction histories of increasing operational complexity\([36](https://arxiv.org/html/2608.17843#bib.bib42);[39](https://arxiv.org/html/2608.17843#bib.bib3);[41](https://arxiv.org/html/2608.17843#bib.bib20)\)\. Generative models accordingly target either constrained sketches, as in Vitruvion\([28](https://arxiv.org/html/2608.17843#bib.bib2)\), or CAD construction sequences, as in DeepCAD and SkexGen\([39](https://arxiv.org/html/2608.17843#bib.bib3);[40](https://arxiv.org/html/2608.17843#bib.bib4)\)\. More recent systems such as Text2CAD, CAD\-Llama, CADmium, STEP\-LLM, and ReCAD leverage language models to generate CAD sequences or executable code\([15](https://arxiv.org/html/2608.17843#bib.bib5);[18](https://arxiv.org/html/2608.17843#bib.bib23);[6](https://arxiv.org/html/2608.17843#bib.bib21);[29](https://arxiv.org/html/2608.17843#bib.bib22);[17](https://arxiv.org/html/2608.17843#bib.bib24)\)\.

Across these lines of work, evaluation has centered on task\-level outcomes such as conditional constraint prediction, CAD reconstruction, validity, and generation quality\. We instead examine whether frozen general\-purpose LLMs encode constraint structure and how that encoding relates to generation, prediction sensitivity to activation restoration, and behavioral control\.

#### Representation analysis and intervention\.

A common approach to representation analysis is linear probing, which tests what information is linearly decodable from frozen hidden states\([1](https://arxiv.org/html/2608.17843#bib.bib7);[2](https://arxiv.org/html/2608.17843#bib.bib9)\)\. Beyond linguistic attributes, probing has identified linearly decodable world states in game\-playing models\([19](https://arxiv.org/html/2608.17843#bib.bib35);[22](https://arxiv.org/html/2608.17843#bib.bib33)\)and spatial and temporal information in LLMs\([8](https://arxiv.org/html/2608.17843#bib.bib34)\)\. These findings are consistent with the linear representation hypothesis, which proposes that features are organized along directions in representation space\([23](https://arxiv.org/html/2608.17843#bib.bib36)\)\. Under this view, linear probing provides a natural tool for examining structured information in model representations\.

However, high linear\-probe accuracy does not by itself show that the relevant representation was learned through pretraining\. Instead, it may reflect probe capacity, architectural bias, or information already present in the input\([11](https://arxiv.org/html/2608.17843#bib.bib8);[24](https://arxiv.org/html/2608.17843#bib.bib10)\)\. Prior work addresses these alternatives using shuffled\-label tasks and selectivity\([11](https://arxiv.org/html/2608.17843#bib.bib8)\), random encoders\([35](https://arxiv.org/html/2608.17843#bib.bib37)\), and comparisons with simpler input representations\([10](https://arxiv.org/html/2608.17843#bib.bib38)\)\. Following these principles, we use shuffled\-label and random\-initialization controls and introduce a pure\-input baseline for serialized geometry\.

Beyond identifying what probes can extract, prior work has developed interventions that test whether internal representations participate in model behavior\([26](https://arxiv.org/html/2608.17843#bib.bib39)\)\. Activation patching tests how interventions on intermediate states affect model outputs\([33](https://arxiv.org/html/2608.17843#bib.bib11);[21](https://arxiv.org/html/2608.17843#bib.bib12)\)\. Related intervention methods have been used to identify behaviorally relevant circuits\([34](https://arxiv.org/html/2608.17843#bib.bib15);[3](https://arxiv.org/html/2608.17843#bib.bib14)\)\. Activation addition and representation engineering instead modify internal states to steer model behavior\([32](https://arxiv.org/html/2608.17843#bib.bib40);[42](https://arxiv.org/html/2608.17843#bib.bib41)\)\.These methods test complementary aspects of representational function\. Patching asks whether restoring an activation at a selected position affects the prediction, whereas steering asks whether modifying that activation can control the output\. We combine these interventions with decoding and generation to distinguish linear accessibility, behavioral expression, activation\-level influence, and controllability\.

## 3Method

We evaluate frozen LLMs using geometry\-only CAD serializations, three matched prediction tasks, and two activation\-level interventions\. Figure[1](https://arxiv.org/html/2608.17843#S1.F1)summarizes the evaluation pipeline\.

#### Data and labels\.

From the SketchGraphs training split\([27](https://arxiv.org/html/2608.17843#bib.bib1)\), we derive pairwise entity\-relation labels \(P1\) and sketch\-level constraint\-status labels \(P2\)\. ForP1, ordered entity pairs\(i,j\)\(i,j\)are assigned to eight classes: seven pairwise relations \(Coincident,Parallel,Perpendicular,Tangent,Equal,Midpoint, andConcentric\) andNoConstraint, sampled from unconstrained pairs in the same sketch\. We excludeHorizontalandVerticalbecause they are primarily unary\. ForP2, sketches are labeled as under\-, well\-, or over\-constrained according to whether the degree\-of\-freedom count returned by SketchGraphsget\_sequence\_dofis positive, zero, or negative\. We construct stratified, class\-balanced subsets of up to 15k pairs per P1 class and 12k sketches per P2 class \(minimum 500 per class\) to address class imbalance, particularly the limited number of well\- and over\-constrained sketches\. The same sampled subsets are reused across models with a fixed seed\. The P2 labels are heuristic rather than solver\-verified and may therefore misclassify redundant constraint sets\.

#### Tasks\.

We organize the evaluation into two decoding tasks and one matched behavioral task\.P1tests whether pairwise constraints are linearly decodable from concatenated entity representations,𝐱i​j=\[𝐡i;𝐡j\]\\mathbf\{x\}\_\{ij\}=\[\\mathbf\{h\}\_\{i\};\\mathbf\{h\}\_\{j\}\], using eight\-class classification\.P2tests whether sketch\-level constraint status is linearly decodable from the mean\-pooled entity representation,𝐡¯=\|ℰ\|−1​∑e∈ℰ𝐡e\\bar\{\\mathbf\{h\}\}=\|\\mathcal\{E\}\|^\{\-1\}\\sum\_\{e\\in\\mathcal\{E\}\}\\mathbf\{h\}\_\{e\}, using three\-class classification\. Whereas P1 and P2 examine linear accessibility at local and global levels,P3provides a behavioral counterpart to P1 by asking the frozen LLM to complete the templateConstraint\(Eii,Ejj\)=…=\\ldotsthrough forced choice over the same eight classes\. Using the same held\-out pairs, label space, and macro\-F1 metric enables a matched comparison between P1 decodability and P3 generation\. The main P3 evaluation is zero\-shot, and Appendix[G](https://arxiv.org/html/2608.17843#A7)reports a four\-shot control across five exemplar sets\.

#### Serialization and label exclusion\.

Each sketch is serialized as plain text containing entity types and numeric parameters, such as line endpoints and circle centers and radii\. AllEdgeOpannotations are omitted, so constraint labels cannot be read directly from the input\. Each entity’s character span is mapped to its corresponding token span for pooling\.

#### Representation extraction\.

We pass each serialized sketch through a frozen decoder\-only LLM\. At each sampled layer, we mean\-pool the token\-level hidden states within each entity span to obtain𝐡e\\mathbf\{h\}\_\{e\}\. To compare architectures with different depths, we extract representations from eight evenly spaced relative\-depth locations rather than shared absolute layer indices\. The resulting representations are stored in FP16 shards, using the same balanced examples across all models\.

#### Probes and controls\.

For P1 and P2, we train a separateℓ2\\ell\_\{2\}\-regularized logistic regression at each layer on balanced data, using a class\-stratified 75/25 split\. The main P1 split is performed at the entity\-pair level rather than the sketch level\. A five\-seed sketch\-level group\-split check on Qwen2\.5\-3B yields comparable performance, suggesting that sketch overlap does not explain the P1 result \(Appendix[D](https://arxiv.org/html/2608.17843#A4)\)\. To isolate the contribution of pretraining, we compare pretrained representations with same\-architecture randomly initialized models and a pure\-input baseline\. Shuffled\-label probes control for probe memorization, while entity\-count controls test whether P2 performance can be explained by sketch size\.

#### Shuffled\-order control\.

To test whether P1 relies on entity position, we randomly permute the entity order within each serialization while preserving entity identities, geometry, and labels, then repeat representation extraction and probing for every model\. The resulting performance change measures sensitivity to serialization order \(Section[5\.2](https://arxiv.org/html/2608.17843#S5.SS2)\)\.

#### Activation\-level influence and steerability\.

To measure prediction sensitivity to activations at the probed entity position, we perform activation patching on Qwen2\.5\-3B and Llama\-3\.1\-8B\. We corrupt entityii’s input embedding with Gaussian noise and restore its clean hidden state at each tested layer\([33](https://arxiv.org/html/2608.17843#bib.bib11);[21](https://arxiv.org/html/2608.17843#bib.bib12);[9](https://arxiv.org/html/2608.17843#bib.bib13)\)\. Restoration rate is the fraction of corruption\-informative examples for which patching recovers the clean prediction\. We test layers at four\-layer intervals and additionally include each model’s P1 decodability peak\. Distractor specificity measures whether restoring entityiileaves the prediction for an unrelated pair\(k,m\)\(k,m\)unchanged\. To determine whether this activation\-level influence can support targeted control, we add class mean\-difference vectors at the strongest nontrivial restoration layer \(layer 4 for both models\), withα∈\{0\.5,1,2,4,8\}\\alpha\\in\\\{0\.5,1,2,4,8\\\}\. We measure flip\-to\-target rates and compare them with matched random\-direction controls across 10 seeds\.

## 4Experimental Setup

#### Models\.

We evaluate six frozen decoder\-only LLMs spanning multiple scales and families: Qwen2\.5\-0\.5B, 1\.5B, 3B, and 7B\([25](https://arxiv.org/html/2608.17843#bib.bib32)\), Mistral\-7B\([14](https://arxiv.org/html/2608.17843#bib.bib16)\), and Llama\-3\.1\-8B\([7](https://arxiv.org/html/2608.17843#bib.bib6)\), each compared against a randomly initialized model of the same architecture\. All six are run through the identical extraction and probing protocol described in Section 3, drawing from the same balanced subsets with a fixed sampling seed reused across models, so that cross\-model differences reflect the models themselves rather than sampling noise\.

#### Metrics\.

We evaluate probe performance using macro\-F1, with uniform\-class reference levels of0\.1250\.125for P1 and0\.3330\.333for P2\. We additionally report selectivity,F1,task−F1,shuffledF\_\{1,\\mathrm\{task\}\}\-F\_\{1,\\mathrm\{shuffled\}\}\([11](https://arxiv.org/html/2608.17843#bib.bib8)\), to control for probe memorization\.

To quantify whether pretraining contributes differently to local and global decodability, we define a dissociation index \(DI\)\. For each architecture, we first identify the layerℓ∗\\ell^\{\*\}at which the pretrained model achieves its highest P1 macro\-F1:

ℓ∗=arg⁡maxℓ​F1,preP​1​\(ℓ\)\.\\ell^\{\*\}=\\arg\\max\_\{\\ell\}F\_\{1,\\mathrm\{pre\}\}^\{P1\}\(\\ell\)\.\(1\)We then hold this layer fixed for all four quantities entering DI\. In particular, pretrained and random initialized P2 performance are evaluated at the P1\-selected layerℓ∗\\ell^\{\*\}rather than at an independently selected P2 peak\. The corresponding random\-initialized P1 performance is also read atℓ∗\\ell^\{\*\}\. We define

DI=\\displaystyle\\mathrm\{DI\}=\{\}\(F1,preP​1​\(ℓ∗\)−F1,randP​1​\(ℓ∗\)\)\\displaystyle\\left\(F\_\{1,\\mathrm\{pre\}\}^\{P1\}\(\\ell^\{\*\}\)\-F\_\{1,\\mathrm\{rand\}\}^\{P1\}\(\\ell^\{\*\}\)\\right\)\(2\)−\(F1,preP​2​\(ℓ∗\)−F1,randP​2​\(ℓ∗\)\)\.\\displaystyle\-\\left\(F\_\{1,\\mathrm\{pre\}\}^\{P2\}\(\\ell^\{\*\}\)\-F\_\{1,\\mathrm\{rand\}\}^\{P2\}\(\\ell^\{\*\}\)\\right\)\.Here,pre\\mathrm\{pre\}andrand\\mathrm\{rand\}denote pretrained and same\-architecture randomly initialized models, respectively\. A positive DI indicates that pretraining improves P1 more than P2 under this common\-layer comparison\. Subtracting the corresponding random\-init baselines partially controls for architectural and dimensional differences across model families and scales\.

P3 uses the same held\-out entity pairs, eight\-class label set, and macro\-F1 metric as P1, enabling a matched comparison between supervised linear decodability and forced\-choice generation\. Unless otherwise noted, we report single\-split F1 estimates using seed 0\. Reported 95% confidence intervals use 1,000 bootstrap resamples\. For DI, uncertainty from the P1 and P2 components is combined in quadrature rather than estimated with a direct paired bootstrap\.

## 5Results

### 5\.1Constraint Information Is Linearly Decodable

Both P1 and P2 are linearly decodable from the hidden states of all six trained models\. Peak P1 macro\-F1 ranges from 0\.714 to 0\.734, well above the 0\.125 chance level \(Table[1](https://arxiv.org/html/2608.17843#S5.T1)\)\. Selectivity remains high at 0\.593–0\.606, indicating that this performance is not explained by shuffled\-label memorization\. Across architectures, P1 decodability rises rapidly in early layers and remains high across a broad depth range \(Figure[2](https://arxiv.org/html/2608.17843#S5.F2)\)\. P2 reaches similarly high peak macro\-F1 values of 0\.719–0\.732, although Section[5\.2](https://arxiv.org/html/2608.17843#S5.SS2)shows that most of this performance is already available without pretraining\.

Figure 2:P1 macro\-F1 across relative depth for all six models and controls\. Orange brackets show the drop under entity\-order shuffling; green marks show the remaining gain over random initialization\.
### 5\.2P1 Benefits More from Pretraining Than P2

The controls reveal different sources of P1 and P2 probe performance\. For P1, macro\-F1 increases from 0\.359 with pure\-input features to 0\.549–0\.598 with random\-init representations and 0\.714–0\.734 with pretrained representations\. Pretraining therefore improves macro\-F1 over random initialization by 0\.127 to 0\.185\. Shuffling entity order lowers P1 macro\-F1 by between 0\.10 and 0\.13, showing that position provides a substantial shortcut\. Even after shuffling, pretrained models outperform their random\-init counterparts by between 0\.026 and 0\.075 \(Figure[2](https://arxiv.org/html/2608.17843#S5.F2)\)\. Thus, positional cues explain some, but not all, of P1’s gain from pretraining\.

P2 relies much less on pretraining\. At the P1\-selected layerℓ∗\\ell^\{\*\}, pretraining improves P2 macro\-F1 over random initialization by only 0\.037 to 0\.048, compared with 0\.127 to 0\.185 for P1 \(Table[1](https://arxiv.org/html/2608.17843#S5.T1)\)\. Thus, most of P2’s decodability is already present without learned weights\. The conclusion is unchanged when P2 is evaluated at its own peak, where the gain remains 0\.037–0\.047\. A logistic regression using only the number of entities reaches 0\.419, showing that sketch size provides some signal but cannot explain the full P2 performance\. Overall, pretraining contributes substantially more to P1 decodability than to P2, even after accounting for the tested positional shortcut\.

Table 1:Probe performance across models\. P1 columns and P1 layer/total report each checkpoint’s own P1 peak\.†P2 is evaluated at the corresponding pretrained model’s P1 peak,ℓ∗\\ell^\{\*\}\. DI evaluates all four trained and random\-init terms at this pretrainedℓ∗\\ell^\{\*\}\(Eq\.[2](https://arxiv.org/html/2608.17843#S4.E2)\)\. Net gain‡is shuffled\-order pretrained P1 macro\-F1 minus the random\-init P1 peak\.
### 5\.3The Dissociation Holds Across Scale and Architecture

We next test whether the P1–P2 dissociation extends beyond a single model\. Within the Qwen2\.5 family, P1 macro\-F1 ranges from 0\.714 to 0\.725 and P2 from 0\.719 to 0\.732, with no monotonic improvement as model size increases \(Table[1](https://arxiv.org/html/2608.17843#S5.T1)\)\. DI nevertheless remains positive at every scale, ranging from 0\.106 to 0\.167, although it also varies non\-monotonically\. Scaling therefore has no consistent effect on either task or on their relative pretraining gains\.

The dissociation also holds across model families\. At comparable model sizes, Qwen2\.5, Llama, and Mistral achieve similar raw macro\-F1 on P1 and P2, but P1 selectivity is consistently higher \(0\.597 to 0\.602 versus 0\.394 to 0\.400\)\. More directly, DI is positive for all six models, with every 95% confidence interval excluding zero\. The selected P1 peaks span layers 10 to 21, indicating that the pattern is not tied to a shared absolute depth\. Chance\-normalizedDInorm\\mathrm\{DI\}\_\{\\mathrm\{norm\}\}also remains positive across all models \(0\.107 to 0\.178; Table[4](https://arxiv.org/html/2608.17843#A2.T4), Appendix[B\.2](https://arxiv.org/html/2608.17843#A2.SS2)\)\. Together, these results show that the dissociation is stable across the tested scales, architectures, and chance normalization\.

### 5\.4Generation Falls Short of Decodability

We compare P3 forced\-choice generation with P1 decoding on the same held\-out pairs and eight\-class label set\. Across all six models, P3 macro\-F1 is substantially lower than P1 probe performance, with gaps ranging from 0\.460 to 0\.700 \(Table[2](https://arxiv.org/html/2608.17843#S5.T2)\)\. Thus, information that is linearly decodable is not reliably expressed in the model’s own predictions\.

Table 2:Matched P1 probe and P3 generation macro\-F1 on the same held\-out pairs and eight\-class label set \(chance=0\.125\{\}=0\.125\)\.Failure modes differ across architectures \(Figure[3](https://arxiv.org/html/2608.17843#S5.F3)\)\. Mistral\-7B predictsCoincidentfor 99\.8% of examples, producing an almost complete single\-class collapse\. Qwen2\.5\-7B instead predicts all eight classes and achieves non\-trivial accuracy on several, yielding the highest P3 macro\-F1 of 0\.259 despite having P1 performance similar to the other models\. Full per\-class results are reported in Appendix[C\.1](https://arxiv.org/html/2608.17843#A3.SS1)\.

Content\-free prompts reveal class preferences aligned with these outputs\. The dominant blank\-prompt class matches the dominant real\-task prediction for both Mistral\-7B and Qwen2\.5\-7B, with maximum prior probabilities of 0\.345 and 0\.250, respectively\. These controls are consistent with prior bias contributing to P3 behavior, but do not by themselves determine how much of the real\-task distribution it explains\. Prompting also accounts for only part of the gap on Qwen2\.5\-3B\. Four\-shot prompting increases mean macro\-F1 from 0\.081 to0\.138±0\.0130\.138\\pm 0\.013across five exemplar sets, but remains 0\.576 below the P1 probe score \(Appendix[G](https://arxiv.org/html/2608.17843#A7)\)\. Few\-shot examples improve generation, but P3 still performs far below the linear probe\.

![Refer to caption](https://arxiv.org/html/2608.17843v1/figures/p3_heatmap_fixed.png)Figure 3:P3 predicted\-class distributions across six models\. Boxes mark the dominant class, and dashes denote exact zeros\. Per\-class accuracies are reported in Appendix[C\.1](https://arxiv.org/html/2608.17843#A3.SS1)\.
### 5\.5Activation\-Level Influence and Steerability

Activation patching reveals an early but transient influence at the probed entity position\. On Qwen2\.5\-3B, restoration peaks at layer 4 with a rate of 0\.781 \[0\.722, 0\.846\] and falls to zero by layer 16, before the P1 decodability peak at layer 21\. Llama\-3\.1\-8B shows the same pattern, peaking at layer 4 with a restoration rate of 0\.876 \[0\.821, 0\.924\] and reaching zero by layer 12, before its decodability peak at layer 14\. Neither model shows a later resurgence through the deepest tested layer\. In contrast, decodability reaches a broad plateau by approximately layers 5 to 9 and persists after restoration effects disappear\. Full layerwise results are shown in Figure[7](https://arxiv.org/html/2608.17843#A5.F7)and Appendix[E\.1](https://arxiv.org/html/2608.17843#A5.SS1)\.

At the layer 4 restoration peak, distractor specificity is 0\.798 for Qwen2\.5\-3B and 0\.847 for Llama\-3\.1\-8B\. Restoration is therefore largely, but not perfectly, specific to the patched entity at the layer where its effect is strongest\. Specificity reaches 1\.0 only at later layers, after the restoration rate has fallen to zero\.

We next ask whether this activation\-level influence can be harnessed for targeted control\. At the same layer, mean\-difference steering produces no target\-class flips at any tested strength for either P1 or P2 on either model, a result reproduced in two independent runs\. On Qwen2\.5\-3B, the mean\-difference direction changes about four times as many labels as matched random directions atα=8\\alpha=8, but none of these changes reach the intended target class\. On Llama\-3\.1\-8B, label changes do not exceed the random baseline \(Appendix[E\.2](https://arxiv.org/html/2608.17843#A5.SS2)\)\. Thus, activation restoration can influence predictions without providing reliable targeted control\.

## 6Discussion

### 6\.1Representational Dissociation

Our main finding is that the linear accessibility of geometric constraint information in frozen LLMs does not ensure its expression in generation, continued influence at the probed entity position, or controllability through mean\-difference steering\. The clearest layerwise contrast is between decodability and activation restoration\. Decodability persists across a broad depth range, whereas restoration effects at the probed entity position are early and transient\. A layer may therefore retain linearly recoverable information even after predictions are no longer sensitive to restoring the activation at that position\.

One possible explanation is that later computation routes the relevant information to other token positions or into distributed representations, leaving a readable trace at the original entity position after dependence on that position has diminished\. The difference between patching and steering may also reflect intervention scope\. Patching restores the full activation vector at the tested position, whereas mean\-difference steering modifies a single direction that may not capture the combination of features used by the model\. These explanations remain hypotheses that require circuit\-level analysis\.

### 6\.2Implications for Interpretability

These results delimit the conclusions supported by each interpretability method\. Linear probing demonstrates that information is recoverable, not that it is behaviorally used\. Activation patching shows that intervening on an activation at a tested position can affect the output, but does not identify which decodable feature mediates that effect\. Similarly, failure under mean\-difference steering rules out that intervention direction, not every possible form of control\. Representational claims should therefore distinguish recoverability, behavioral expression, activation\-level influence, and control, while remaining scoped to the interventions actually tested\.

### 6\.3Practical Implications

For practical CAD systems, high probe scores are not enough to establish reliable geometric reasoning\. P2 is highly decodable even without pretraining, and strong P1 decodability does not translate into accurate constraint predictions\. Systems that require valid outputs should therefore not rely on LLM representations alone and may need explicit validity supervision, structured state tracking, or solver\-based verification\.

### 6\.4Limitations

Our conclusions apply only to geometry\-only inputs from SketchGraphs\. P1 excludes the primarily unaryHorizontalandVerticalconstraints\. P2 labels are derived from a heuristic Grübler\-style DOF count rather than solver verification\. Because the heuristic does not verify constraint independence, sketches with redundant constraints may receive incorrect labels\. Moreover, P2 tests only coarse DOF status, not other aspects of global geometry such as consistency, solvability, or redundancy\.

Our uncertainty estimates do not cover all sources of experimental variation\. The main P1 and P2 results use one fixed data split, each architecture has one random\-init checkpoint, and order shuffling and P3 sampling use fixed seeds\. P1 is split by entity pair rather than by sketch\. Although a five\-seed sketch\-level split check gives comparable results on Qwen2\.5\-3B \(Appendix[D](https://arxiv.org/html/2608.17843#A4)\), it was not repeated across all architectures\. In addition, DI confidence intervals capture uncertainty from resampling the evaluation examples, but not variation from data splitting, initialization, or layer selection\. They are also approximate because the independently bootstrapped component uncertainties are combined in quadrature rather than obtained by directly bootstrapping DI\.

The behavioral and intervention analyses cover a narrower range of settings than the decoding experiments\. P3 uses a single forced\-choice formulation, and the four\-shot control focuses on Qwen2\.5\-3B\. Performance may vary with other label formulations, prompts, or exemplar choices\. The intervention experiments cover two backbones\. Patching is evaluated on corruption\-informative pairs \(n=169n\{=\}169and145145\), while steering examines mean\-difference directions up toα=8\\alpha=8\. These results may vary with the model, corruption scheme, intervention direction, or token position\. Finally, our residual\-stream interventions operate at an aggregate level and do not localize the effects to specific attention heads, MLPs, or neurons\.

### 6\.5Future Work

The present results leave open how broadly the observed dissociation extends beyond SketchGraphs\. Testing other geometric corpora and CAD\-native models such as Vitruvion\([28](https://arxiv.org/html/2608.17843#bib.bib2)\)would establish its generality\. Our Fusion 360 Gallery experiment provides initial cross\-dataset evidence for P1 \(Appendix[F](https://arxiv.org/html/2608.17843#A6)\), but the dataset lacks matched three\-class labels for P2\. A broader evaluation of global reasoning will therefore require solver\-derived validity signals that improve on the heuristic Grübler\-style labels used here\. The behavioral and intervention analyses could likewise be extended across additional backbones, verbalizers, prompting strategies, patching designs, and steering directions\. Building on these experiments, circuit\-level analysis could examine whether the gaps among decodability, generation, and control arise from information routing, distributed computation, or context\-dependent representations\.

## 7Conclusion

We introduced a four\-part audit of linear decodability, forced\-choice generation, activation\-level influence, and behavioral steerability in frozen LLMs\. By evaluating these properties separately under matched conditions, the framework provides a controlled way to identify where representational evidence does and does not translate into model behavior\. Across six models, pretraining contributes substantially more to pairwise constraint decoding than to sketch\-level DOF classification, for which randomly initialized models already achieve high probe performance\. P3 generation remains well below supervised P1 decoding, and a four\-shot control on Qwen2\.5\-3B narrows but does not close this gap\. On the two backbones tested with interventions, activation restoration affects predictions primarily at early layers and vanishes at the patched entity position while decodability persists\. Mean\-difference steering at the strongest restoration layer produces no targeted class flips\.

The results show that decodability, behavioral expression, activation\-level influence, and control are empirically distinct: a high probe score demonstrates that information can be linearly recovered, but provides limited evidence that a model will express, use, or respond to interventions on that information\. Parametric CAD makes these distinctions directly testable, and our audit offers a framework for separating failures to encode constraint structure from failures to act on what is already encoded\.

## References

- Alain and Bengio \(2017\)G\. Alain and Y\. BengioUnderstanding intermediate layers using linear classifier probes\.InICLR Workshop,Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p1.1)\.
- Belinkov and Glass \(2019\)Y\. Belinkov and J\. GlassAnalysis methods in neural language processing: a survey\.Transactions of the Association for Computational Linguistics7,pp\. 49–72\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p1.1)\.
- Conmyet al\.\(2023\)A\. Conmy, A\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-AlonsoTowards automated circuit discovery for mechanistic interpretability\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1)\.
- Fanet al\.\(2026\)F\. Fan, J\. Ni, F\. Sang, X\. Yin, Y\. Liu, R\. Tong, M\. Tang, and P\. DuTraceCAD: Trace\-Guided Repair for Agentic CAD Generation\.External Links:[Link](https://www.semanticscholar.org/paper/TraceCAD%3A-Trace-Guided-Repair-for-Agentic-CAD-Fan-Ni/b06fff0aa9a211df31e448d4548c02d324c84d22)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p2.1)\.
- González\-Lluchet al\.\(2019\)C\. González\-Lluch, P\. Company, M\. Contero, D\. Pérez\-López, and J\. D\. CambaOn the effects of the fix geometric constraint in 2D profiles on the reusability of parametric 3D CAD models\.International Journal of Technology and Design Education29\(4\),pp\. 821–841\(en\)\.External Links:ISSN 0957\-7572, 1573\-1804,[Link](http://link.springer.com/10.1007/s10798-018-9458-z),[Document](https://dx.doi.org/10.1007/s10798-018-9458-z)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p1.1)\.
- Govindarajanet al\.\(2025\)P\. Govindarajan, D\. Baldelli, J\. Pathak, Q\. Fournier, and S\. ChandarCADmium: Fine\-Tuning Code Language Models for Text\-Driven Sequential CAD Design\.External Links:[Link](https://arxiv.org/abs/2507.09792),[Document](https://dx.doi.org/10.48550/ARXIV.2507.09792)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian,et al\.The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p3.1),[§4](https://arxiv.org/html/2608.17843#S4.SS0.SSS0.Px1.p1.1)\.
- Gurnee and Tegmark \(2024\)W\. Gurnee and M\. TegmarkLanguage Models Represent Space and Time\.arXiv\.Note:arXiv:2310\.02207 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2310.02207),[Document](https://dx.doi.org/10.48550/arXiv.2310.02207)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p1.1)\.
- Hannaet al\.\(2023\)M\. Hanna, O\. Liu, and A\. VariengienHow does GPT\-2 compute greater\-than?: interpreting mathematical abilities in a pre\-trained language model\.InAdvances in Neural Information Processing Systems,Cited by:[§3](https://arxiv.org/html/2608.17843#S3.SS0.SSS0.Px7.p1.1)\.
- Hewittet al\.\(2021\)J\. Hewitt, K\. Ethayarajh, P\. Liang, and C\. ManningConditional probing: measuring usable information beyond a baseline\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 1626–1639\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.122/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.122)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p2.1)\.
- Hewitt and Liang \(2019\)J\. Hewitt and P\. LiangDesigning and interpreting probes with control tasks\.InEmpirical Methods in Natural Language Processing,Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p2.1),[§4](https://arxiv.org/html/2608.17843#S4.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2026\)T\. Hu, J\. Ai, L\. Wen, X\. Li, S\. Zou, S\. Li, N\. Deng, X\. Cai, H\. Zhou, P\. Cai, D\. Fu, Y\. Yang, H\. Zhang, B\. Shi, and X\. YangIterCAD: An Iterative Multimodal Agent for Visually\-Grounded CAD Generation and Editing\.External Links:[Link](https://arxiv.org/abs/2606.13368),[Document](https://dx.doi.org/10.48550/ARXIV.2606.13368)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p2.1)\.
- Huanget al\.\(2025\)I\. Huang, Y\. Bao, K\. Truong, H\. Zhou, C\. Schmid, L\. Guibas, and A\. FathiFirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement\.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 13466–13476\.External Links:[Link](https://ieeexplore.ieee.org/document/11094419/),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.01257)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p1.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p3.1),[§4](https://arxiv.org/html/2608.17843#S4.SS0.SSS0.Px1.p1.1)\.
- Khanet al\.\(2024\)M\. S\. Khan, S\. Sinha, T\. U\. Sheikh, D\. Stricker, S\. A\. Ali, and M\. Z\. AfzalText2CAD: generating sequential CAD designs from beginner\-to\-expert level text prompts\.InAdvances in Neural Information Processing Systems,pp\. 7552–7579\.Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p2.1),[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Leuet al\.\(2013\)M\. C\. Leu, H\. A\. ElMaraghy, A\. Y\.C\. Nee, S\. K\. Ong, M\. Lanzetta, M\. Putz, W\. Zhu, and A\. BernardCAD model based virtual assembly simulation, planning and training\.CIRP Annals62\(2\),pp\. 799–822\(en\)\.External Links:ISSN 00078506,[Link](https://linkinghub.elsevier.com/retrieve/pii/S0007850613001959),[Document](https://dx.doi.org/10.1016/j.cirp.2013.05.005)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025a\)J\. Li, Y\. Luo, Y\. Lou, and X\. ZhouReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision\-Language Models\.External Links:[Link](https://arxiv.org/abs/2512.06328),[Document](https://dx.doi.org/10.48550/ARXIV.2512.06328)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Liet al\.\(2025b\)J\. Li, W\. Ma, X\. Li, Y\. Lou, G\. Zhou, and X\. ZhouCAD\-Llama: Leveraging Large Language Models for Computer\-Aided Design Parametric 3D Model Generation\.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 18563–18573\.External Links:[Link](https://ieeexplore.ieee.org/document/11094068/),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.01730)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p2.1),[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Liet al\.\(2024\)K\. Li, A\. K\. Hopkins, D\. Bau, F\. Viégas, H\. Pfister, and M\. WattenbergEmergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task\.arXiv\.Note:arXiv:2210\.13382 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2210.13382),[Document](https://dx.doi.org/10.48550/arXiv.2210.13382)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)F\. Liu, H\. Zhou, F\. Hao, and L\. YangEmbodied CAD: Solver\-Grounded LLM Agents for Parametric B\-Rep Assembly Modeling\.External Links:[Link](https://arxiv.org/abs/2606.31252),[Document](https://dx.doi.org/10.48550/ARXIV.2606.31252)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p2.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1),[§3](https://arxiv.org/html/2608.17843#S3.SS0.SSS0.Px7.p1.1)\.
- Nandaet al\.\(2023\)N\. Nanda, A\. Lee, and M\. WattenbergEmergent Linear Representations in World Models of Self\-Supervised Sequence Models\.arXiv\.Note:arXiv:2309\.00941 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2309.00941),[Document](https://dx.doi.org/10.48550/arXiv.2309.00941)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p1.1)\.
- Parket al\.\(2024\)K\. Park, Y\. J\. Choe, and V\. VeitchThe Linear Representation Hypothesis and the Geometry of Large Language Models\.arXiv\.Note:arXiv:2311\.03658 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2311.03658),[Document](https://dx.doi.org/10.48550/arXiv.2311.03658)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p1.1)\.
- Pimentelet al\.\(2020\)T\. Pimentel, J\. Pimentel, H\. Velioglu, M\. Wich, and R\. CotterellInformation\-theoretic probing for linguistic structure\.InAssociation for Computational Linguistics,pp\. 4609–4619\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p2.1)\.
- Qwenet al\.\(2025\)Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui,et al\.Qwen2\.5 Technical Report\.arXiv\.Note:arXiv:2412\.15115 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2412.15115),[Document](https://dx.doi.org/10.48550/arXiv.2412.15115)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p3.1),[§4](https://arxiv.org/html/2608.17843#S4.SS0.SSS0.Px1.p1.1)\.
- Ravichanderet al\.\(2021\)A\. Ravichander, Y\. Belinkov, and E\. HovyProbing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,P\. Merlo, J\. Tiedemann, and R\. Tsarfaty \(Eds\.\),Online,pp\. 3363–3377\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1)\.
- Seffet al\.\(2020\)A\. Seff, Y\. Ovadia, W\. Zhou, and R\. P\. AdamsSketchGraphs: a large\-scale dataset for modeling relational geometry in CAD\.InICML Workshop on Object\-Oriented Learning,Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p1.1),[§1](https://arxiv.org/html/2608.17843#S1.p3.1),[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1),[§3](https://arxiv.org/html/2608.17843#S3.SS0.SSS0.Px1.p1.1)\.
- Seffet al\.\(2022\)A\. Seff, W\. Zhou, N\. Richardson, and R\. P\. AdamsVitruvion: a generative model of parametric CAD sketches\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1),[§6\.5](https://arxiv.org/html/2608.17843#S6.SS5.p1.1)\.
- Shiet al\.\(2026\)X\. Shi, J\. Ding, X\. Zhao, S\. Zhan, P\. Mohapatra, D\. Quispe, K\. Welbeck, J\. Cao, W\. Chen, P\. Guo, and Q\. ZhuSTEP\-LLM: Generating CAD STEP Models from Natural Language with Large Language Models\.External Links:[Link](https://arxiv.org/abs/2601.12641),[Document](https://dx.doi.org/10.48550/ARXIV.2601.12641)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Singhet al\.\(2023\)I\. Singh, V\. Blukis, A\. Mousavian, A\. Goyal, D\. Xu, J\. Tremblay, D\. Fox, J\. Thomason, and A\. GargProgPrompt: Generating Situated Robot Task Plans using Large Language Models\.2023 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 11523–11530\.External Links:[Link](https://ieeexplore.ieee.org/document/10161317/),[Document](https://dx.doi.org/10.1109/ICRA48891.2023.10161317)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p1.1)\.
- Thierryet al\.\(2011\)S\. E\. B\. Thierry, P\. Schreck, D\. Michelucci, C\. Fünfzig, and J\. GénevauxExtensions of the witness method to characterize under\-, over\- and well\-constrained geometric constraint systems\.Computer\-Aided Design43\(10\),pp\. 1234–1249\.External Links:ISSN 0010\-4485,[Link](https://www.sciencedirect.com/science/article/pii/S0010448511001606),[Document](https://dx.doi.org/10.1016/j.cad.2011.06.018)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p1.1)\.
- Turneret al\.\(2024\)A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmidSteering Language Models With Activation Engineering\.arXiv\.Note:arXiv:2308\.10248 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2308.10248),[Document](https://dx.doi.org/10.48550/arXiv.2308.10248)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1)\.
- Viget al\.\(2020\)J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, Y\. Singer, and S\. ShieberInvestigating gender bias in language models using causal mediation analysis\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4548–4561\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1),[§3](https://arxiv.org/html/2608.17843#S3.SS0.SSS0.Px7.p1.1)\.
- Wanget al\.\(2023\)K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. SteinhardtInterpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1)\.
- Wieting and Kiela \(2019\)J\. Wieting and D\. KielaNo Training Required: Exploring Random Encoders for Sentence Classification\.arXiv\.Note:arXiv:1901\.10444 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/1901.10444),[Document](https://dx.doi.org/10.48550/arXiv.1901.10444)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p2.1)\.
- Williset al\.\(2021\)K\. D\. D\. Willis, Y\. Pu, J\. Luo, H\. Chu, T\. Du, J\. G\. Lambourne, A\. Solar\-Lezama, and W\. MatusikFusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences\.arXiv\(en\)\.Note:arXiv:2010\.02392 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2010.02392),[Document](https://dx.doi.org/10.48550/arXiv.2010.02392)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p3.1),[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Wuet al\.\(2023\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: Enabling Next\-Gen LLM Applications via Multi\-Agent Conversation\.InarXiv preprint arXiv:2308\.08155,External Links:[Link](https://www.semanticscholar.org/paper/AutoGen%3A-Enabling-Next-Gen-LLM-Applications-via-Wu-Bansal/9ea0757c750ab1222a7442d3485a74d1c526b04c)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p1.1)\.
- Wuet al\.\(2025\)R\. Wu, W\. Su, and J\. LiaoChat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models\.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 23690–23700\.External Links:[Link](https://ieeexplore.ieee.org/document/11092564/),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.02206)Cited by:[§1](https://arxiv.org/html/2608.17843#S1.p1.1)\.
- Wuet al\.\(2021\)R\. Wu, C\. Xiao, and C\. ZhengDeepCAD: a deep generative network for computer\-aided design models\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 6772–6782\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Xuet al\.\(2022\)X\. Xu, K\. D\. D\. Willis, J\. G\. Lambourne, C\. Cheng, P\. K\. Jayaraman, and Y\. FurukawaSkexGen: autoregressive generation of CAD construction sequences with disentangled codebooks\.InInternational Conference on Machine Learning,pp\. 24698–24724\.Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Zhouet al\.\(2023\)S\. Zhou, T\. Tang, and B\. ZhouCADParser\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence,Guide Proceedings,pp\. 1804–1812\.External Links:[Link](https://dl.acm.org/doi/10.24963/ijcai.2023/200),[Document](https://dx.doi.org/10.24963/ijcai.2023/200)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px1.p2.1)\.
- Zouet al\.\(2025\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. HendrycksRepresentation Engineering: A Top\-Down Approach to AI Transparency\.arXiv\.Note:arXiv:2310\.01405 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2310.01405),[Document](https://dx.doi.org/10.48550/arXiv.2310.01405)Cited by:[§2](https://arxiv.org/html/2608.17843#S2.SS0.SSS0.Px2.p3.1)\.

## Appendix ATask and Label Details

### A\.1Geometry\-Only Serialization and Matched Tasks

Figure 4 illustrates the input and labeling protocol using one toy sketch\. The geometric entities are serialized for the frozen LLM, while theEdgeOprelation is used only as a supervision or evaluation target and never appears in the model input\.

![Refer to caption](https://arxiv.org/html/2608.17843v1/figures/serialization_audit_headers_v2.png)Figure 4:Worked example of the geometry\-only protocol\. A source sketch \(a\) is serialized with allEdgeOplabels excluded \(b\) and used to construct the matched P1–P3 evaluation targets \(c\)\.
### A\.2P1/P2 Constraint\-Type Gallery

Figure[5](https://arxiv.org/html/2608.17843#A1.F5)shows the eight P1 classes: seven pairwiseEdgeOprelations and a sampledNoConstraintclass\. It also illustrates the three P2 DOF labels: under\-, well\-, and over\-constrained\.

![Refer to caption](https://arxiv.org/html/2608.17843v1/figures/p1_constraint_gallery.png)Figure 5:P1 pairwise constraint classes and P2 structural DOF labels in SketchGraphs\. Highlighted entities indicate the pair probed in P1;EdgeOptokens are excluded from the LLM input\.
### A\.3P2 Label Quality Checks

We assess two potential concerns with the heuristic DOF\-status labels used for P2\.

#### Entity\-count baseline\.

Because under\-constrained sketches tend to contain fewer entities, sketch size may provide a classification shortcut\. A logistic regression using entity count alone achieves 0\.419 macro\-F1, above the uniform\-class reference of 0\.333 but well below both random\-init \(0\.679–0\.691\) and pretrained probes \(0\.719–0\.732\)\. Sketch size therefore explains some, but not most, of the observed P2 performance\.

#### Constraint satisfaction\.

We also test whether the constraints assigned to each sketch can be satisfied simultaneously\. The test is passed by 100% of well\-constrained sketches and 97\.5% of over\-constrained sketches\. This provides evidence of geometric feasibility, but does not validate constraint independence\. Redundant constraints may remain jointly satisfiable while removing fewer independent degrees of freedom than the heuristic assumes\. A full validation would require solver\-based rank analysis of the constraint Jacobian \(Section[6\.5](https://arxiv.org/html/2608.17843#S6.SS5)\)\.

## Appendix BRepresentation and Split Controls

### B\.1Random\-Init Controls

Table[3](https://arxiv.org/html/2608.17843#A2.T3)reports the independently selected P1 and P2 peaks from one random initialization of each architecture\. Hidden states are extracted and probed using the same pipeline as for the pretrained models\. These task\-specific peaks are descriptive controls and are not used to compute DI, which evaluates all four component scores at the pretrained model’s P1\-selected layerℓ∗\\ell^\{\*\}\.

Table 3:Task\-specific peak macro\-F1 scores for the random\-init controls\. The final column reports the layer of the P1 peak\.Random\-init P1 peaks range from 0\.549 to 0\.598, well below the pretrained range of 0\.714–0\.734\. Random\-init P2 peaks, however, reach 0\.681–0\.695, substantially above the pure\-input baseline of 0\.380 and only modestly below the pretrained range of 0\.719–0\.732\. Across all six architectures, random\-init representations therefore reproduce P2 performance much more closely than P1 performance\.

### B\.2Chance\-Normalized Dissociation Index

The main analysis computes DI on the raw macro\-F1 scale \(Section[5\.3](https://arxiv.org/html/2608.17843#S5.SS3)\)\. Because P1 and P2 have different uniform\-class reference levels, we repeat the analysis after normalizing each score by its headroom above the corresponding reference:

gt​\(F1\)=F1−ct1−ct,cP​1=18,cP​2=13\.g\_\{t\}\(F\_\{1\}\)=\\frac\{F\_\{1\}\-c\_\{t\}\}\{1\-c\_\{t\}\},\\qquad c\_\{P1\}=\\frac\{1\}\{8\},\\quad c\_\{P2\}=\\frac\{1\}\{3\}\.\(3\)
For taskt∈\{P​1,P​2\}t\\in\\\{P1,P2\\\}, the raw and normalized pretraining gains are defined as

Δt\\displaystyle\\Delta^\{t\}=F1,pret​\(ℓ∗\)−F1,randt​\(ℓ∗\),\\displaystyle=F\_\{1,\\mathrm\{pre\}\}^\{t\}\(\\ell^\{\*\}\)\-F\_\{1,\\mathrm\{rand\}\}^\{t\}\(\\ell^\{\*\}\),\(4\)Δnormt\\displaystyle\\Delta\_\{\\mathrm\{norm\}\}^\{t\}=Δt1−ct\.\\displaystyle=\\frac\{\\Delta^\{t\}\}\{1\-c\_\{t\}\}\.The chance\-normalized dissociation index is then

DInorm=ΔnormP​1−ΔnormP​2\.\\mathrm\{DI\}\_\{\\mathrm\{norm\}\}=\\Delta\_\{\\mathrm\{norm\}\}^\{P1\}\-\\Delta\_\{\\mathrm\{norm\}\}^\{P2\}\.\(5\)The reference terms cancel within each pretrained–random\-init contrast, leaving each gain rescaled by its task\-specific headroom\.

Table 4:Raw and chance\-normalized DI at each model’s P1\-selected layerℓ∗\\ell^\{\*\}\.As shown in Table[4](https://arxiv.org/html/2608.17843#A2.T4), chance\-normalized DI remains positive for all six models and closely tracks the raw DI\. The P1–P2 dissociation therefore cannot be explained by the tasks’ different uniform\-class reference levels\.

## Appendix CP3 Diagnostics

### C\.1P3 Per\-Class Accuracy

Figure[3](https://arxiv.org/html/2608.17843#S5.F3)reports how frequently each model predicts each class\. Here, Figure[6](https://arxiv.org/html/2608.17843#A3.F6)reports accuracy conditional on the true class, while Table[5](https://arxiv.org/html/2608.17843#A3.T5)presents both quantities\. In each table cell, the first value is the fraction of all examples predicted as that class, and the second is accuracy among examples whose true label is that class\.

![Refer to caption](https://arxiv.org/html/2608.17843v1/p3_accuracy_heatmap.png)Figure 6:P3 accuracy by model and true constraint class\. Dashes denote exact zeros\.Table 5:P3 predicted\-class frequency and per\-class accuracy\. Each cell reports the fraction of all predictions assigned to that class, followed by accuracy among examples with that true label\. Column headers give the true\-class counts \(n=2,000n=2\{,\}000total\)\. Bold marks each model’s most frequently predicted class\. Values are rounded to two decimals; dashes denote exact zeros\.The models show different generation failure modes\. Mistral\-7B collapses almost entirely toCoincidentand succeeds only when that label is correct\. Qwen2\.5\-7B distributes its predictions across more classes and shows uneven class\-specific accuracy:Parallelis predicted most often, whereas Tangent is predicted most accurately\.

### C\.2P3 Content\-Free Prior Control

To measure class preferences in the absence of geometric content, Section[5\.4](https://arxiv.org/html/2608.17843#S5.SS4)evaluates five content\-free entity\-index templates\. Table[6](https://arxiv.org/html/2608.17843#A3.T6)reports the mean probability assigned to each class for Mistral\-7B and Qwen2\.5\-7B\.

Table 6:Mean P3 class probabilities across five content\-free templates\. Bold marks the highest\-probability class for each model\.The content\-free preferences align with the dominant real\-task predictions\. Mistral\-7B assigns the highest prior probability toCoincident, while Qwen2\.5\-7B assigns the highest probability toParallel, followed closely byTangent\. This alignment suggests that class priors contribute to the observed prediction patterns\. However, because the control reports probability mass whereas the real\-task analysis reports argmax frequencies, it does not determine how much of those patterns is explained by prior bias\.

## Appendix DP1 Sketch\-Level Split Check

To test whether cross\-partition sketch overlap inflates P1 performance, we repeat the Qwen2\.5\-3B probe at its selected layer \(ℓ∗=21\\ell^\{\*\}=21\) usingGroupShuffleSplit, which assigns all entity pairs from a sketch to the same partition\. Across five split seeds, the sketch\-level split achieves a macro\-F1 of0\.700±0\.0020\.700\\pm 0\.002, compared with0\.699±0\.0090\.699\\pm 0\.009for the original pair\-level split \(mean±\\pmstandard deviation\)\. These averages differ from the single\-split estimates in the main results\. The similar performance indicates that sketch overlap does not materially affect the P1 result in this setting\.

## Appendix EIntervention Details

### E\.1P1 Activation\-Patching Layer Grid

We evaluate activation patching at four\-layer intervals and additionally include each model’s P1 decodability peak\. The analysis contains 169 corruption\-informative pairs for Qwen2\.5\-3B and 145 for Llama\-3\.1\-8B\. Figure[7](https://arxiv.org/html/2608.17843#A5.F7)reports restoration rates with 95% confidence intervals from 1,000 bootstrap resamples over pairs\. Layer 0, where restoration directly reverses the embedding corruption, serves as a sanity check and is excluded when selecting the strongest nontrivial restoration layer\.

For Qwen2\.5\-3B, restoration is highest at layer 4 \(0\.781 \[0\.722, 0\.846\]\), declines at layers 8 \(0\.675 \[0\.609, 0\.746\]\) and 12 \(0\.604 \[0\.533, 0\.675\]\), and reaches zero by layer 16, before the P1 decodability peak at layer 21\. Llama\-3\.1\-8B shows the same pattern\. Restoration decreases from 0\.876 \[0\.821, 0\.924\] at layer 4 to 0\.759 \[0\.690, 0\.821\] at layer 8, then remains at zero from layer 12 onward, including at its decodability peak at layer 14\. In both models, restoration at the patched entity position therefore disappears before peak decodability\.

![Refer to caption](https://arxiv.org/html/2608.17843v1/figures/causal_decode_overlay.png)Figure 7:Layerwise P1 restoration rates and decodability for Qwen2\.5\-3B and Llama\-3\.1\-8B\. Annotations mark the strongest nontrivial restoration layer and each model’s P1 decodability peak\. Shaded bands show 95% bootstrap confidence intervals\.
### E\.2Steering at the Restoration Peak

We evaluate steering at layer 4, the strongest nontrivial restoration layer for both models\. Class mean\-difference directions are compared with matched random directions overα∈\{0\.5,1,2,4,8\}\\alpha\\in\\\{0\.5,1,2,4,8\\\}using 200 examples and 10 random\-direction seeds\. Each experiment is repeated in two independent runs\.

Mean\-difference steering produces no target\-class flips at anyα\\alphafor either task or model\. Target\-flip rates under random directions also remain at or below 0\.1%\. Table[7](https://arxiv.org/html/2608.17843#A5.T7)reports the less restrictive P1 label\-change rate, which counts any change in prediction, whether or not it reaches the intended class\.

Table 7:P1 label\-change rates at layer 4 under mean\-difference and matched random\-direction interventions\. Cells report ranges across two independent runs\.For Qwen2\.5\-3B, mean\-difference directions cause more label changes than random directions whenα≥2\\alpha\\geq 2, but none reach the intended class\. For Llama\-3\.1\-8B, their label\-change rates remain at or below the random baseline\. The observed changes therefore do not provide evidence of reliable targeted steering\.

## Appendix FCross\-Dataset P1 Check

We apply the P1 probing protocol to Fusion 360 Gallery reconstruction data \(r1\.0\.1\) using Qwen2\.5\-3B and 13,600 balanced entity pairs \(1,700 per class\)\. This check covers P1 only because Fusion 360 Gallery does not provide matched three\-class DOF\-status labels for P2\. Macro\-F1 peaks at 0\.643 at layer 26 \(72\.2%72\.2\\%relative depth\) and changes little between layers 21 and 31\. This broad intermediate\-to\-late plateau is consistent with the layerwise pattern observed on SketchGraphs\.

Table 8:P1 macro\-F1 and selectivity across sampled Qwen2\.5\-3B layers on Fusion 360 Gallery\.
## Appendix GP3 Few\-Shot Prompting Control

To assess the sensitivity of P3 to prompt format, we evaluate four\-shot prompting on Qwen2\.5\-3B using five independently sampled exemplar sets\. Each prompt contains one labeled example from each of four sampled classes, followed by the same eight\-way forced\-choice task used in the zero\-shot evaluation\. The evaluation pairs and scoring procedure remain unchanged\.

Across the five exemplar sets, four\-shot prompting increases mean macro\-F1 from the zero\-shot score of 0\.081 to0\.138±0\.0130\.138\\pm 0\.013\(mean±\\pmstandard deviation\), with individual scores ranging from 0\.117 to 0\.157\. Mean accuracy is0\.218±0\.0110\.218\\pm 0\.011\. Performance therefore varies with exemplar selection, although every tested set improves over zero\-shot prompting\. Even the best four\-shot result remains 0\.557 below the P1 linear\-probe score of 0\.714, while the gap at the four\-shot mean is 0\.576\. Demonstrations improve the forced\-choice readout, but leave most of the gap to supervised linear decodability unresolved\.