Syntax vs. Semantics: How Transformers Learn Deep Dependencies

# Syntax vs. Semantics: How Transformers Learn Deep Dependencies
Source: [https://arxiv.org/html/2608.26139](https://arxiv.org/html/2608.26139)
###### Abstract

Large Language Models demonstrate remarkable syntactic fluency, yet the optimization dynamics governing their acquisition of deep semantic dependencies remain poorly understood\. We propose a mechanistic framework that models this learning process as a competition between Surface Statistics and Deep Semantics\. Our theoretical analysis identifies a “Gradient Starvation” phenomenon where the error signals for sparse semantic dependencies are actively suppressed during early optimization\. This suppression impedes the learning of structural reasoning and causes its emergence to manifest as a sudden phase transition\. Furthermore, this framework offers a mechanistic basis for the effectiveness of Chain\-of\-Thought \(CoT\) strategies\. By externalizing intermediate reasoning steps into concrete tokens, CoT effectively bypasses the suppression regime inherent to implicit reasoning\. We validate these findings across scales ranging from toy transformers to production models \(Llama\-3\.1\-8B, Qwen2\.5\-Coder\-7B\)\. Finally, guided by this theory, we propose a topology\-aligned contrastive objective that explicitly rectifies the gradient geometry\. Experiments on variable binding tasks demonstrate that our method achieves an improvement that is over 2× larger than that obtained via standard cross\-entropy fine\-tuning\. Code will be publicly available at:[https://github\.com/jr\-zhao/Deep\-Dependencies/tree/main](https://github.com/jr-zhao/Deep-Dependencies/tree/main)\.

Machine Learning, ICML

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.26139v1/x1.png)

Figure 1:Gradient Competition Dynamics\. Left: High\-curvature syntax \(orange\) starves the semantic signal\. Right: The model aligns with the deep topological dependency \(blue\)\.The hallmark of general intelligence is not merely surface\-level fluency, but the ability to acquire and exploitDeep Dependenciesthat extend beyond local patterns\. Such dependencies require models to recover latent relational structure from context, a capability long argued to be central to systematic generalization\(Fodor and Pylyshyn,[1988](https://arxiv.org/html/2608.26139#bib.bib39); Marcus,[1998](https://arxiv.org/html/2608.26139#bib.bib40)\)\. While recent work has demonstrated that Transformer\-based models can eventually capture these abstract dependencies\(Vaswaniet al\.,[2017](https://arxiv.org/html/2608.26139#bib.bib44); Olssonet al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib9)\), the optimization dynamics governing their emergence remain poorly understood\. In practice, models reliably master local, high\-frequency structure early in training, yet only later acquire deep dependency reasoning, often via an abrupt qualitative transition rather than gradual improvement\(Weiet al\.,[2022a](https://arxiv.org/html/2608.26139#bib.bib41); Poweret al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib42); Davieset al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib43); Wuet al\.,[2025](https://arxiv.org/html/2608.26139#bib.bib47)\)\.

We argue that this phenomenon is not architectural but dynamical\. The operators required for deep dependency reasoning are structurally attainable, existing as latent linear subspaces within standard Transformer geometry\(Dhayalkar,[2025](https://arxiv.org/html/2608.26139#bib.bib46); Boix\-Adseraet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib45)\)\. However, accessing these subspaces during optimization is non\-trivial\. We propose a mechanistic framework that characterizes training as a competition betweenSurface Statistics\(syntax\) andDeep Semantics\(latent dependency structure\)\. During early optimization, high\-curvature gradients induced by frequent local patterns dominate the loss landscape, a phenomenon known as “Gradient Starvation”\(Pezeshkiet al\.,[2021](https://arxiv.org/html/2608.26139#bib.bib48)\)\. These gradients suppress weaker, low\-curvature signals associated with long\-range dependencies, effectively masking semantic learning until a critical phase transition occurs\.

The tension between syntactic structure and semantic content has been widely studied, either through architectures that explicitly separate the two\(Russinet al\.,[2019](https://arxiv.org/html/2608.26139#bib.bib52); Felhiet al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib53); Caucheteuxet al\.,[2021](https://arxiv.org/html/2608.26139#bib.bib54)\), the injection of structural priors such as dependency trees\(Baiet al\.,[2021](https://arxiv.org/html/2608.26139#bib.bib55); Gonget al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib56); Zhouet al\.,[2020](https://arxiv.org/html/2608.26139#bib.bib57); Kalyanpuret al\.,[2020](https://arxiv.org/html/2608.26139#bib.bib58)\), or probing analyses that question whether syntactic competence entails semantic generalization\(Weissweileret al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib59); Ahujaet al\.,[2025](https://arxiv.org/html/2608.26139#bib.bib60); Allemanet al\.,[2021](https://arxiv.org/html/2608.26139#bib.bib61)\)\. However, these approaches largely treat syntax and semantics as static representational properties or architectural design choices\. In contrast, our work models them as competitors in the gradient landscape, providing a mechanistic interpretation for why syntactic structure tends to be learned earlier and how it may interfere with the acquisition of deep dependencies during training\.

This framework yields two central predictions\. First, it explains the hierarchical emergence of reasoning: optimization pressure forces attention heads to prioritize recovery of coarse relational topology before resolving fine\-grained values, leading to a characteristic staged development of dependency circuits\. Second, it provides a causal account of the effectiveness of Chain\-of\-Thought \(CoT\) prompting\(Weiet al\.,[2022b](https://arxiv.org/html/2608.26139#bib.bib49)\)\. By externalizing intermediate states, CoT alters the learning geometry by injecting additional gradient pathways, partially bypassing the starvation regime that constrains implicit reasoning\.

To empirically validate these dynamics, we adopt a multi\-stage experimental strategy\. Because natural language lacks unambiguous ground truth for latent dependency structure, we primarily use source code and Abstract Syntax Trees \(ASTs\) as a controlled proxy for semantic topology\(Allamaniset al\.,[2017](https://arxiv.org/html/2608.26139#bib.bib50); Chen,[2021](https://arxiv.org/html/2608.26139#bib.bib51)\)\. We trace the emergence of dependency circuits from controlled setting toy models to intermediate checkpoints of Pythia\(Bidermanet al\.,[2023b](https://arxiv.org/html/2608.26139#bib.bib30)\), and finally demonstrate the effectiveness of our topology\-aligned intervention on production\-scale models, including Qwen2\.5\-Coder\-7B\(Huiet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib35); Yanget al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib36)\)and Llama\-3\.1\-8B\(Dubeyet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib37)\)\.

## 2Mechanistic Characterization: The Dynamics of Semantic Emergence

In this section, we provide a theoretical framework for the emergence of Semantic Binding Circuits \(or Contextual Dependency Circuits\)\. We formulate the problem as the competition between Surface Statistics \(Syntax\) and Deep Semantics \(latent dependency structure\)\. We demonstrate that this competition is governed by a signal\-to\-noise ratio \(alignment ratio\) dynamic, where high\-curvature syntactic features initially dominate the optimization dynamics through gradient starvation, suppressing effective updates along semantic directions and delaying the emergence of systematic reasoning until a critical phase transition occurs\.

### 2\.1Problem Setting: Surface Structure vs\. Deep Binding

Let sequence𝒮=\{x1,…,xT\}\\mathcal\{S\}=\\\{x\_\{1\},\\dots,x\_\{T\}\\\}be the input\. We distinguish two functional dependencies:Syntax \(ℱs​y​n\\mathcal\{F\}\_\{syn\}\), comprising local, high\-frequency patterns \(e\.g\., separators\); andSemantics \(ℱs​e​m\\mathcal\{F\}\_\{sem\}\), comprising long\-range dependencies that recover the computational topology \(specifically, mapping a Query token to its informational predecessor in the program graph\)\.

### 2\.2The Geometry of Competition: Gradient Starvation

###### Assumption 2\.1\(Spectral Disparity\)\.

We assume optimization pressure is uneven\. The syntactic subspace𝒰s​y​n\\mathcal\{U\}\_\{syn\}spans eigenvectors with large curvatureλs​y​n\\lambda\_\{syn\}, while the semantic subspace𝒰s​e​m\\mathcal\{U\}\_\{sem\}has smaller curvatureλs​e​m≪λs​y​n\\lambda\_\{sem\}\\ll\\lambda\_\{syn\}\. The initial residualr0r\_\{0\}has non\-trivial energy in𝒰s​y​n\\mathcal\{U\}\_\{syn\}\.

This assumption is empirically verified in App\.[A\.1](https://arxiv.org/html/2608.26139#A1.SS1)\.

###### Proposition 2\.2\(Gradient Starvation via Spectral Bias\)\.

Consider the local quadratic approximation of the lossℒ​\(θ\)≈ℒ​\(θ∗\)\+12​\(θ−θ∗\)T​H​\(θ−θ∗\)\\mathcal\{L\}\(\\theta\)\\approx\\mathcal\{L\}\(\\theta^\{\*\}\)\+\\frac\{1\}\{2\}\(\\theta\-\\theta^\{\*\}\)^\{T\}H\(\\theta\-\\theta^\{\*\}\)\. Lete=θ−θ∗e=\\theta\-\\theta^\{\*\}denote the parameter residual\. The gradientg≈H​eg\\approx Hecan be decomposed in the Hessian eigenbasis\{\(λk,vk\)\}\\\{\(\\lambda\_\{k\},v\_\{k\}\)\\\}as:g=∑kλk​\(vkT​e\)​vkg=\\sum\_\{k\}\\lambda\_\{k\}\(v\_\{k\}^\{T\}e\)v\_\{k\}, where scalar\(vkT​e\)\(v\_\{k\}^\{T\}e\)is the projection of the residual onto thekk\-th eigenvector\. Assumingλs​y​n≫λs​e​m\\lambda\_\{syn\}\\gg\\lambda\_\{sem\}and that the residual has comparable average projection magnitudes on both subspaces, the gradient magnitude is dominated by syntactic components:

‖gs​y​n‖2‖gs​e​m‖2≈λs​y​nλs​e​m⋅ms​y​nms​e​m≫1,\\frac\{\\\|g\_\{syn\}\\\|\_\{2\}\}\{\\\|g\_\{sem\}\\\|\_\{2\}\}\\;\\approx\\;\\frac\{\\lambda\_\{syn\}\}\{\\lambda\_\{sem\}\}\\cdot\\sqrt\{\\frac\{m\_\{syn\}\}\{m\_\{sem\}\}\}\\;\\;\\gg\\;\\;1,\(1\)wherems​y​nm\_\{syn\}andms​e​mm\_\{sem\}are the effective dimensions of the two subspaces\. Consequently, the update direction−g\-\\\!gis dominated by the syntactic component, and the optimization makes much slower progress along semantic directions, yielding a masking \(gradient\-starvation\) effect forℱs​e​m\\mathcal\{F\}\_\{sem\}\(Proof: App\.[C\.1](https://arxiv.org/html/2608.26139#A3.SS1)\)\.

### 2\.3Suppression Mechanism: Softmax Saturation

We analyze how early syntactic convergence impedes semantic learning\. Letss​e​ms\_\{sem\}be the logit for the correct semantic token, andAs​y​n≈1A\_\{syn\}\\approx 1be the attention weight on a distracting syntactic token\.

###### Lemma 2\.3\(Gradient Suppression Factor\)\.

Letss​e​ms\_\{sem\}be the attention score for the correct semantic token andAs​e​mA\_\{sem\}be its softmax weight\. In the saturated regime dominated by syntax \(As​y​n≈1A\_\{syn\}\\approx 1\), we haveAs​e​m≈0A\_\{sem\}\\approx 0\. The gradient of the lossℒ\\mathcal\{L\}with respect to the scoress​e​ms\_\{sem\}is given by:

∂ℒ∂ss​e​m\\displaystyle\\frac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{sem\}\}=∑k∂ℒ∂Ak​∂Ak∂ss​e​m\\displaystyle=\\sum\\nolimits\_\{k\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}\\frac\{\\partial A\_\{k\}\}\{\\partial s\_\{sem\}\}\(2\)=As​e​m​\(∂ℒ∂As​e​m−∑kAk​∂ℒ∂Ak\)\.\\displaystyle=A\_\{sem\}\\big\(\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{sem\}\}\-\\sum\\nolimits\_\{k\}A\_\{k\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}\\big\)\.
Thus, the magnitude scales linearly with the attention weight:\|∂ℒ∂ss​e​m\|∝As​e​m\\left\|\\dfrac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{sem\}\}\\right\|\\propto A\_\{sem\}\.

###### Theorem 2\.4\(The Vanishing Gradient Barrier\)\.

Under the top\-1 domination regime where a single syntactic scoress​y​ns\_\{syn\}dominates the softmax normalizer, we haveAs​e​m≈exp⁡\(−\(ss​y​n−ss​e​m\)\)A\_\{sem\}\\approx\\exp\(\-\(s\_\{syn\}\-s\_\{sem\}\)\)\. When the model is in the “Syntactic Phase” \(As​y​n→1A\_\{syn\}\\to 1\), the effective learning signal for the semantic component vanishes exponentially with the score differencess​y​n−ss​e​ms\_\{syn\}\-s\_\{sem\}\. Under bounded∂ℒ∂Ak\\dfrac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}, we have

\|∂ℒ∂ss​e​m\|=O​\(e−\(ss​y​n−ss​e​m\)\)→0\(Derivation:Appendix[C\.2](https://arxiv.org/html/2608.26139#A3.SS2)\)\.\\left\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{sem\}\}\\right\|=O\\\!\\left\(e^\{\-\(s\_\{syn\}\-s\_\{sem\}\)\}\\right\)\\to 0\\quad\\text\{\\small\(Derivation:Appendix\\ref\{app:softmax\_saturation\}\)\.\}
Implication:This multiplicative suppression creates a Gradient Barrier\. Even if the definition token contains useful information, the signal is attenuated\. Semantic learning is thus effectively stalled until the syntactic confidenceAs​y​nA\_\{syn\}degrades \(e\.g\., via conflicting patterns or regularization\), raisingAs​e​mA\_\{sem\}above a learnable threshold\.

### 2\.4Emergence via Subspace Alignment

Once the attention head escapes saturation, the interaction matrixWQ​K=WQT​WKW\_\{QK\}=W\_\{Q\}^\{T\}W\_\{K\}aligns with the underlying semantic topology\.

###### Assumption 2\.5\(Error\-Noise Orthogonality\)\.

We decompose each token embedding asx=μ\+δx=\\mu\+\\delta, whereμ\\mulies in a low\-dimensional signal subspace andδ\\deltais zero\-mean context noise\.

###### Assumption 2\.6\(Signal Decomposition\)\.

We additionally assume𝔼​\[δ\]=0\\mathbb\{E\}\[\\delta\]=0and𝔼​\[δj​δiT\]≈0\\mathbb\{E\}\[\\delta\_\{j\}\\delta\_\{i\}^\{T\}\]\\approx 0for unrelated token pairs\. Letγj​i=∂ℒ∂sj​i\\gamma\_\{ji\}=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{ji\}\}be the scalar error signal flowing back to the attention score\. We assume that under the data distribution𝒟\\mathcal\{D\}, the backward error signalγ\\gammais uncorrelated with the forward context noiseδ\\deltainherent in the embeddings:

𝔼𝒟​\[γj​i⋅δi\]≈0\.\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\\gamma\_\{ji\}\\cdot\\delta\_\{i\}\]\\approx 0\.

This assumption is empirically verified in App\.[A\.3](https://arxiv.org/html/2608.26139#A1.SS3)\.

###### Proposition 2\.7\(Hebbian Reconstruction of Dependency Topology\)\.

The attention score is the bilinear formsj​i=xjT​WQ​K​xis\_\{ji\}=x\_\{j\}^\{T\}W\_\{QK\}x\_\{i\}\. The gradient of the loss with respect to the interaction matrixWQ​KW\_\{QK\}accumulates as the outer product of inputs:

∇WQ​Kℒ=∑j,iγj​i​xj​xiT,where​γj​i=∂ℒ∂sj​i\.\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}=\\sum\_\{j,i\}\\gamma\_\{ji\}x\_\{j\}x\_\{i\}^\{T\},\\quad\\text\{where \}\\gamma\_\{ji\}=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{ji\}\}\.Substituting the signal decompositionx=μ\+δx=\\mu\+\\delta\(Assumption[2\.5](https://arxiv.org/html/2608.26139#S2.Thmtheorem5)\) and invoking the orthogonality assumption\(justification in App\.[C\.3](https://arxiv.org/html/2608.26139#A3.SS3)\), the expected update direction becomes:

𝔼𝒟​\[∇WQ​Kℒ\]≈𝔼​\[γj​i\]⋅\(μu​s​e​μs​r​cT\)\.\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}\]\\approx\\mathbb\{E\}\[\\gamma\_\{ji\}\]\\cdot\(\\mu\_\{use\}\\mu\_\{src\}^\{T\}\)\.\(3\)This confirms that the optimizer naturally pushesWQ​KW\_\{QK\}to align with the rank\-1 outer productμu​s​e​μs​r​cT\\mu\_\{use\}\\mu\_\{src\}^\{T\}, which represents the topological edge in the computation graph\.

### 2\.5Phase Transitions and the Critical Threshold

To quantify the transition, we define theAlignment Ratioρ​\(t\)\\rho\(t\)\. We define the ideal topological operatorMs​e​mM\_\{sem\}as the rank\-1 outer product encoding the dependency edge:

Ms​e​m=μu​s​e​μs​r​cT\.M\_\{sem\}=\\mu\_\{use\}\\mu\_\{src\}^\{T\}\.Note thatMs​e​mM\_\{sem\}is generally not symmetric \(as usage≠\\neqsource\)\. We defineρ​\(t\)\\rho\(t\)as the squared cosine similarity between the current gradient update and this target operator:

ρ​\(t\)=⟨∇WQ​Kℒ,Ms​e​m⟩F2‖∇WQ​Kℒ‖F2⋅‖Ms​e​m‖F2\+ϵ\.\\rho\(t\)=\\frac\{\\langle\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\},M\_\{sem\}\\rangle\_\{F\}^\{2\}\}\{\\\|\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}\\\|\_\{F\}^\{2\}\\cdot\\\|M\_\{sem\}\\\|\_\{F\}^\{2\}\+\\epsilon\}\.\(4\)
This metricρ​\(t\)∈\[0,1\]\\rho\(t\)\\in\[0,1\]measures geometrical alignment\(threshold derivation: App\.[C\.4](https://arxiv.org/html/2608.26139#A3.SS4)\)\. Based on these thresholds, we identify three distinct phases:

##### Phase I: Syntactic Dominance \(ρ​\(t\)<τl​o​w\\rho\(t\)<\\tau\_\{low\}\)\.

Due to the high curvature of syntactic features \(Prop\.[2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)\), gradients are dominated by the syntactic subspace, leading the model to preferentially fit local n\-gram patterns\. Under this regime, the vanishing\-gradient barrier induced by softmax saturation \(Thm\.[2\.4](https://arxiv.org/html/2608.26139#S2.Thmtheorem4)\) suppresses effective updates along semantic directions, preventing the stable formation of binding circuits\.

##### Phase II: The Crossover \(ρ​\(t\)∈\[τl​o​w,τs​t​a​b​l​e\)\\rho\(t\)\\in\[\\tau\_\{low\},\\tau\_\{stable\}\)\)\.

This is the volatile heuristic phase\. As syntactic loss saturates, the update direction begins to acquire a non\-trivial component along the semantic operatorMs​e​mM\_\{sem\}, causingρ​\(t\)\\rho\(t\)to rise from near\-zero to an intermediate regime\. However, the alignment is not yet stable: competing heuristics and residual syntactic correlations induce high\-variance gradients, preventing consistent consolidation\.

##### Phase III: Systematic Emergence \(ρ​\(t\)≥τs​t​a​b​l​e\\rho\(t\)\\geq\\tau\_\{stable\}\)\.

Whenρ​\(t\)\\rho\(t\)exceeds a stability thresholdτs​t​a​b​l​e\\tau\_\{stable\}, the gradient becomes aligned with the topological operatorμu​s​e​μs​r​cT\\mu\_\{use\}\\mu\_\{src\}^\{T\}\. The interaction matrixWQ​KW\_\{QK\}locks onto this low\-rank\(nearly11\) direction, effectively implementing the copy mechanism\(Empirical verification of rank collapse: App\.[A\.2](https://arxiv.org/html/2608.26139#A1.SS2)\)\.

###### Corollary 2\.8\(Sparsity via Competitive Alignment\)\.

The emergence of binding circuits is constrained by the residual\-stream additivity across heads\. Lete:=∇xℒe:=\\nabla\_\{x\}\\mathcal\{L\}denote the backpropagated error at the residual stream of the layer\. The gradient magnitude for a specific headhhis controlled by the projection of this error onto the semantic subspace:

‖∇WQ​K\(h\)ℒ‖≲C⋅‖Proj𝒰s​e​m​\(e\)‖\.\\\|\\nabla\_\{W^\{\(h\)\}\_\{QK\}\}\\mathcal\{L\}\\\|\\;\\lesssim\\;C\\cdot\\\|\\text\{Proj\}\_\{\\mathcal\{U\}\_\{sem\}\}\(e\)\\\|\.\(5\)Due to initialization variance, a early\-aligned headh∗h^\{\*\}crossesτs​t​a​b​l​e\\tau\_\{stable\}first, effectively reducing‖Proj𝒰s​e​m​\(e\)‖\\\|\\text\{Proj\}\_\{\\mathcal\{U\}\_\{sem\}\}\(e\)\\\|\. While the semantic residual does not vanish, it decreases sufficiently such that for the remaining headsh≠h∗h\\neq h^\{\*\}, the alignment signal is overwhelmed by the high\-curvature syntactic noise \(Prop\.[2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)\)\. Consequently, the effective SNR for lagging heads drops below the critical threshold required to escape the syntactic regime, confining the semantic function to a sparse subset of heads\.

##### Mechanism: The Precedence of Topological Neighbors\.

We predict that the alignment ratioρ​\(t\)\\rho\(t\)for the pointer edge \(i→ji\\to j\) consistently crosses the stability thresholdτs​t​a​b​l​e\\tau\_\{stable\}before the value edge \(i→ki\\to k\)\. The edgei→ji\\to jtypically represents a Type\-Schema relationship \(e\.g\., Operator→\\toOperand, or Attribute→\\toEntity\), which corresponds to a dense, lower\-curvature region of the semantic manifold compared to the sparse\(Visualized via subspace probing in App\.[A\.4](https://arxiv.org/html/2608.26139#A1.SS4)\), high\-frequency map to the specific valuekk\. Mathematically, the projection of the gradient onto the pointer operatorMp​t​r=μi​μjTM\_\{ptr\}=\\mu\_\{i\}\\mu\_\{j\}^\{T\}dominates the projection onto the value operatorMv​a​l=μi​μkTM\_\{val\}=\\mu\_\{i\}\\mu\_\{k\}^\{T\}:

⟨∇ℒ,Mp​t​r⟩F≫⟨∇ℒ,Mv​a​l⟩F⟹ρi→j​\(t\)\>ρi→k​\(t\)\.\\langle\\nabla\\mathcal\{L\},M\_\{ptr\}\\rangle\_\{F\}\\gg\\langle\\nabla\\mathcal\{L\},M\_\{val\}\\rangle\_\{F\}\\implies\\rho\_\{i\\to j\}\(t\)\>\\rho\_\{i\\to k\}\(t\)\.\(6\)Thus, the dynamics force the attention head to first lock onto the intermediate nodejj, establishing the “Pointer Passing” circuit as the primary topological relaxation \(SNR analysis: App\.[C\.5](https://arxiv.org/html/2608.26139#A3.SS5)\)\.

##### Subsequent Integration via Residual Streams\.

Once the connectioni→ji\\to jis established, the residual stream property facilitates the secondary emergence of direct fetching\. The updatexi←xi\+WO​V​xjx\_\{i\}\\leftarrow x\_\{i\}\+W\_\{OV\}x\_\{j\}physically moves the representation of the pointerjjinto the positionii\. For a deeper layerLL, the distance to the valuekkis effectively shortened\. The attention mechanism can now utilize the copied content ofjjto resolve the dependencyj→kj\\to kdirectly from positionii, appearing phenomenologically as a “skip\-connection” to the valuekk\.

### 2\.6Accelerating Semantic Emergence via CoT

We distinguish the mechanistic role of CoT in two regimes:➀ Inference \(State Externalization\):Implicit reasoning requires resolving multi\-hop dependencies \(x→…→yx\\to\\dots\\to y\) in a single pass\. CoT externalizes intermediate states, factorizing the difficult conditionalP​\(y\|x\)P\(y\|x\)into local stepsP​\(z\|x\)​P​\(y\|x,z\)P\(z\|x\)P\(y\|x,z\), reducing the complexity of attention retrieval\.➁ Training \(Gradient Injection\):Supervision on traces introduces a local objectiveℒt​o​t​a​l=ℒy\|z\+ℒz\\mathcal\{L\}\_\{total\}=\\mathcal\{L\}\_\{y\|z\}\+\\mathcal\{L\}\_\{z\}\.

###### Proposition 2\.11\(Parallel Gradient Pathway\)\.

In implicit learning, the gradient∇θℒi​m​p\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{imp\}must backpropagate through the entire downstream computation ofyy\. This signal is vulnerable to attenuation if the downstream dependency is weak or noisy\. CoT supervision injects a local gradient term∇θℒz\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{z\}\. Crucially, this term is independent of downstream correctness \(i\.e\., it does not depend on the gradient flow fromyy\)\. This short\-circuits the long credit assignment chain, ensuring that the binding circuit receives a valid learning signal even when the implicit downstream signal is vanishing or confused\.

## 3Controlled Experiments: Verifying the Signal\-to\-Noise Dynamics

We utilize a small Transformer trained on synthetic code as a controlled setting environment\. While our theoretical framework \(Section[2](https://arxiv.org/html/2608.26139#S2)\) applies to general sequential reasoning, code allows us to precisely decouple Surface Statistics \(syntax tokens like=,;\) from Deep Semantics \(variable dependencies\), a separation often obscured in natural language\. All detailed experiment settings can be seen in App\.[B\.1](https://arxiv.org/html/2608.26139#A2.SS1)\.

The training data is constrained to \(i\) variable assignment \(binding definition\) and \(ii\) arithmetic operations \(binding usage\)\. We design three experiments to empirically verify the theoretical predictions: specifically, the existence of gradient starvation \(Prop\.[2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)\), the release from Softmax Saturation \(Thm[2\.4](https://arxiv.org/html/2608.26139#S2.Thmtheorem4)\), and the causal effect of manipulating the alignment ratioρ​\(t\)\\rho\(t\)\(Remark[2\.9](https://arxiv.org/html/2608.26139#S2.Thmtheorem9)\)\.

### 3\.1Visualizing Gradient Starvation

We first track the training dynamics to validate the gradient starvation hypothesis \([Proposition2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)\)\. We monitor both the loss curve and the gradient norms associated with syntactic tokens \(𝒯s​y​n\\mathcal\{T\}\_\{syn\}\) versus variable tokens \(𝒯s​e​m\\mathcal\{T\}\_\{sem\}\)\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x2.png)

Figure 2:Left: training loss over optimization steps\. Right: average embedding gradient norms for different token groups, where the red curve corresponds to syntactic tokens \(e\.g\., LET, =, ;\) and the blue curve corresponds to variable tokens \(e\.g\., v0, v1, …\)\.As shown in Figure[2](https://arxiv.org/html/2608.26139#S3.F2), the model exhibits a clear two\-stage learning process\. In the early phase, despite large semantic errors reflected by a high loss, gradients on variable tokens \(blue curve\) remain suppressed and comparable to the syntactic baseline, consistent with gradient starvation: under Softmax Saturation \([Section2\.3](https://arxiv.org/html/2608.26139#S2.SS3)\), semantic errors fail to propagate to the embeddings\.

Once syntactic gradients decay and attention patterns destabilize, a sharp phase transition occurs\. The sudden surge in variable gradients signals the release of the Softmax gate, enabling the accumulated semantic error to update the identifier subspace\. This delayed gradient burst indicates active inhibition of the semantic circuit rather than its absence\.

### 3\.2Escaping Softmax Saturation

We examine the mechanism of the phase transition by tracking attention allocation\. We employ a curriculum setup: training on variable assignment until convergence, then introducing addition tasks at step15001500\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x3.png)

Figure 3:Task accuracy and attention distributions during curriculum training\. The green line shows accuracy on variable assignment\(copy\), and the black line shows accuracy on addition after the task switch\. The blue line denotes attention to operand tokens \(e\.g\., variables involved in the calculation\), while the red dotted line denotes attention to syntactic tokens \(e\.g\., LET, =, ;\)\.Figure[3](https://arxiv.org/html/2608.26139#S3.F3)illustrates the behavior predicted by Theorem[2\.4](https://arxiv.org/html/2608.26139#S2.Thmtheorem4)\. Early in training, attention is locked onto syntactic anchors \(red dotted line\), enforcingAs​e​m≈0A\_\{sem\}\\approx 0and suppressing semantic error propagation as predicted by Eq\.[2\.3](https://arxiv.org/html/2608.26139#S2.Thmtheorem3), resulting in low task accuracy\.

As syntactic error diminishes, attention confidence weakens and a crossover occurs: attention to operand tokens \(blue line\) exceeds syntactic attention, unlocking the gradient gate and enabling semantic alignment in the identifier subspace\. This transition coincides with a rapid accuracy increase and is consistently observed in both variable assignment and addition tasks as the alignment ratioρ​\(t\)\\rho\(t\)crosses the critical thresholdτs​t​a​b​l​e∈\(0,1\)\\tau\_\{stable\}\\in\(0,1\)\.

### 3\.3Causal Validation of alignment ratio Thresholds

![Refer to caption](https://arxiv.org/html/2608.26139v1/x4.png)

Figure 4:Accuracy comparison across three conditions: Suppress Syntax \(red solid line\), Control \(blue solid line\), and Noisy Variables \(green dashed line\)\.Finally, we perform targeted interventions to further investigate the role of the alignment ratioρ​\(t\)\\rho\(t\)\. Based on the prescriptive implication in Remark[2\.9](https://arxiv.org/html/2608.26139#S2.Thmtheorem9), we compare a standard Control model against three variants: \(1\) a syntax\-suppressed model, where gradients on syntactic tokens are artificially masked \(equivalent to∇s​y​n→0\\nabla\_\{syn\}\\to 0\); \(2\) a variable\-disturbed model, where noise is injected into variable\-token gradients; and \(3\) a syntax\-free setting, where high\-frequency syntactic markers are removed while preserving the underlying semantic dependency structure\.

The results in Figure[4](https://arxiv.org/html/2608.26139#S3.F4)provide strong empirical support for the gradient\-starvation hypothesis\. When gradients associated with syntactic tokens are suppressed \(red\), the update direction becomes less dominated by non\-semantic components, which increases the alignment ratioρ​\(t\)\\rho\(t\)and accelerates alignment with semantic dependency structure\. Similarly, the syntax\-free setting \(orange\) exits the early optimization plateau substantially earlier, suggesting that syntactic competition itself contributes to delayed dependency learning\. In contrast, injecting noise into variable\-token gradients \(green\) perturbs the semantic alignment direction, reduces the correlation between∇WQ​Kℒ\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}andMs​e​mM\_\{sem\}, lowersρ​\(t\)\\rho\(t\), and consequently delays semantic emergence\.

We additionally replicate the same transition behavior in a controlled natural\-language entity\-binding task, where removing high\-frequency functional tokens similarly accelerates semantic alignment\. This suggests that the proposed mechanism is not limited to code syntax, but may reflect a broader optimization phenomenon in deep dependency learning\.

This double dissociation provides strong evidence that the emergence of binding is closely linked to the competition ratioρ​\(t\)\\rho\(t\)\. It suggests that the syntactic suppression strategy proposed in our theory is a viable method for accelerating reasoning emergence in larger models\.

### 3\.4Summary of Findings

These experiments confirm the three pillars of our theoretical framework: \(1\) Gradient Starvation is a major contributing factor behind the initial learning plateau; \(2\)Softmax Saturationacts as the gatekeeper mechanism; \(3\) The emergence of reasoning is a phase transition governed by the semantic alignment ratioρ​\(t\)\\rho\(t\)\.

Crucially, the success of the Syntax Suppression intervention demonstrates that the learning of structure and reasoning are competitive processes, suggesting that data curation strategies for LLMs should prioritize maximizing logic density over syntactic perfection\.

## 4Mechanistic Experiments: The Dynamics of Emergence and Composition

Having established theoretical bounds in a controlled setting, we extend our analysis to the training dynamics of real\-world LLMs using Pythia\(Bidermanet al\.,[2023b](https://arxiv.org/html/2608.26139#bib.bib30),[a](https://arxiv.org/html/2608.26139#bib.bib31); van der Walet al\.,[2025](https://arxiv.org/html/2608.26139#bib.bib32)\), a family of decoder\-only transformers trained on the Pile\(Gaoet al\.,[2020](https://arxiv.org/html/2608.26139#bib.bib33); Bidermanet al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib34)\)\. Pythia provides dense intermediate checkpoints, enabling a longitudinal analysis of semantic circuit emergence\. Leveraging these checkpoints, we validate our framework across three mechanistic scales: \(i\) alignment phase transitions \(ρ​\(t\)\\rho\(t\)\) in Pythia\-160M to test gradient starvation, \(ii\) the structural priority of pointer formation in Pythia\-1\.4B, and \(iii\) signal propagation in compositional chains, illustrating how CoT bypasses the gradient bottleneck\. All detailed experiment settings can be seen in App\.[B\.2](https://arxiv.org/html/2608.26139#A2.SS2)

### 4\.1Dynamics of the Alignment Ratioρ​\(t\)\\rho\(t\)

To validate the subspace competition hypothesis, we track the geometric alignmentρ​\(t\)\\rho\(t\)in Pythia\-160m\. Since the Transformer architecture parameterizes queries and keys separately, we cannot directly observe the gradient of the composite interaction matrix∇WQ​Kℒ\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}\. Instead, we computeρ​\(t\)\\rho\(t\)using the effective gradient reconstructed via the product rule:∇eff≈\(∇WQℒ\)T​WK\+WQT​\(∇WKℒ\)\\nabla\_\{\\text\{eff\}\}\\approx\(\\nabla\_\{W\_\{Q\}\}\\mathcal\{L\}\)^\{T\}W\_\{K\}\+W\_\{Q\}^\{T\}\(\\nabla\_\{W\_\{K\}\}\\mathcal\{L\}\)\. This metric captures the implicit update to the attention geometry, ensuring mathematical equivalence to our theoretical definition while respecting the model’s factored parameterization\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x5.png)

Figure 5:The evolution of alignment ratioρ​\(t\)\\rho\(t\)in Pythia\-160m reveals a transition from syntactic gradient starvation to Systematic Emergence, punctuated by a volatile crossover phase \(shaded\) that reflects subspace competition\.Figure[5](https://arxiv.org/html/2608.26139#S4.F5)reveals a distinct non\-monotonic transition\. Initially,ρ​\(t\)≈0\\rho\(t\)\\approx 0, providing empirical support for the gradient\-starvation effect \(Prop\.[2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)\) where syntactic noise masks the topological signal\. The transient spike at 5k reflects the volatility of the heuristic phase, where local approximations yield unstable alignment before collapsing\. Subsequently, the monotonic rise from step 20k signifies Systematic Emergence\. This steady consolidation validates Prop\.[2\.7](https://arxiv.org/html/2608.26139#S2.Thmtheorem7), demonstrating that the interaction matrix successfully locks onto the low\-rank semantic operatorMs​e​mM\_\{sem\}after escaping the syntactic basin\.

### 4\.2The Priority of Pointer Passing

To validate the prediction in Remark[2\.10](https://arxiv.org/html/2608.26139#S2.Thmtheorem10), we track the evolution of dependency resolution heads in Pythia\-1\.4B across steps 1k, 20k, and 143k, utilizing transitive identity chains \(Natural Language\) and value assignment chains \(code\) as probes\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x6.png)

Figure 6:We track the attention weights from query tokens \(y\-axis\) to potential antecedents \(x\-axis\) throughout the training of Pythia\-1\.4B\. The probes utilize recursive variable assignments for code \(top; e\.g\.,a = 42; b = a;\) and transitive identity chains for Natural Language \(bottom; e\.g\.,The Alpha is the Bravo\.\), visualizing the emergence of the pointer mechanism\.Figure[6](https://arxiv.org/html/2608.26139#S4.F6)provides strong empirical support for our theory\. In Natural Language, attention heads exhibit a stable diagonal structure throughout training, consistent with stepwise predecessor pointing\. At Step 143k, the amplified first\-column attention further indicates the emergence of multi\-hop aggregation via iterative propagation\.

In contrast, the code modality shows a delayed and shallower formation of this reference mechanism\. We attribute this gap to stronger syntactic curvature in code, which raises the gradient barrier, and to reduced code exposure during pre\-training, which prolongs signal accumulation\.

### 4\.3Mechanistic Validation of CoT: Signal Restoration

To validate the parallel gradient pathway hypothesis \(Prop\.[2\.11](https://arxiv.org/html/2608.26139#S2.Thmtheorem11)\), we compare signal propagation in Implicit versus CoT modes within the training trajectory of Pythia\-160m across steps 1k, 20k, and 143k\. We utilize a gradient saliency probe to compute theSelectivity Ratioℛ\\mathcal\{R\}, defined as the proportion of gradient energy concentrated on the true causal ancestorxs​r​cx\_\{src\}relative to the total input sequence𝒮\\mathcal\{S\}:

ℛ=‖∇xs​r​cℒ‖F‖∇𝒮ℒ‖F\+ϵ\.\\mathcal\{R\}=\\frac\{\\\|\\nabla\_\{x\_\{src\}\}\\mathcal\{L\}\\\|\_\{F\}\}\{\\\|\\nabla\_\{\\mathcal\{S\}\}\\mathcal\{L\}\\\|\_\{F\}\+\\epsilon\}\.\(7\)This metric explicitly quantifies the model’s sensitivity to the true causal ancestor, serving as a proxy for the alignment of the attention mechanism with the underlying computational topology\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x7.png)

Figure 7:We report the log\-ratio of gradient norms between the correct source and a distractor\. Implicit \(Red\): Signal decays with depth, indicating long\-range gradient starvation\. CoT \(Green\): Intermediate supervision restores local gradients, stabilizing long\-horizon learning\.The results in Figure[7](https://arxiv.org/html/2608.26139#S4.F7)demonstrate a divergence in signal dynamics\. Implicit reasoning \(red\) suffers from rapid signal decay as chain depth increases, confirming the downstream insensitivity predicted in Prop\.[2\.11](https://arxiv.org/html/2608.26139#S2.Thmtheorem11)\. In contrast, CoT \(green\) effectively short\-circuits this bottleneck\. By enforcing local stepwise prediction, CoT maintains high gradient selectivity regardless of total depth\. This provides empirical evidence consistent with the hypothesis that externalizing reasoning traces may alleviate gradient starvation by restoring local learning signals\.

## 5Generalization to LLMs

To demonstrate the practical utility of our theoretical framework in realistic scenarios, we design aVariable Cloze Completionexperiment where a specific variable usage in a code snippet is masked \(e\.g\.,return a \+ b; return a \+ <VAR\>;\) and the model must recover the correct identifier based on the context\.

### 5\.1Experiment Settings

#### 5\.1\.1Models\.

We employ two open\-weights models: Qwen2\.5\-Coder\-7B\(Huiet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib35); Yanget al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib36)\)and Llama\-3\.1\-8B\(Dubeyet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib37)\)\. These models represent standard production\-scale architectures, allowing us to verify whether the mechanisms identified in our controlled probes scale to widely used foundation models\.

#### 5\.1\.2Datasets\.

We utilize Python150k and JavaScript150k datasets\(Raychevet al\.,[2016](https://arxiv.org/html/2608.26139#bib.bib38)\)\. Crucially, these datasets provide native ASTs, which enables us to surgically target variable usage nodes for masking rather than relying on random token dropping\. This ensures that the task strictly evaluates the model’s ability to resolve semantic dependencies rather than relying on trivial local n\-grams\.

#### 5\.1\.3Baselines

To better position our method relative to existing structured learning approaches, we additionally compare against several representative baselines:

##### Curriculum Learning

\(Bengioet al\.,[2009](https://arxiv.org/html/2608.26139#bib.bib66)\), where samples are scheduled from easy to hard according to USE–DEF span distance;

##### Linear\-chain CRF

\(Zhenget al\.,[2015](https://arxiv.org/html/2608.26139#bib.bib67)\), which imposes token\-level sequential structural constraints over DEF/USE labels;

##### Supervised Contrastive Learning \(SupCon\)

\(Gunelet al\.,[2020](https://arxiv.org/html/2608.26139#bib.bib68)\), which aligns hidden representations of corresponding USE–DEF pairs;

##### Attention Entropy Regularization

\(Zhaoet al\.,[2019](https://arxiv.org/html/2608.26139#bib.bib69)\), which encourages sharper attention distributions without directional semantic supervision\.

Table 1:Main Results on Variable Cloze Completion\.We report performance across two experimental settings\.Inference\-time Ablation:Basedenotes the original model;Top8andRandomrefer to masking the identified binding heads versus a random selection to verify their functional importance\.Fine\-tuning Methods:SFTrepresents standard cross\-entropy training;CRF,Curriculum,SupCon, andAttndenote the structured learning baselines described in Section 5\.2;Oursutilizes the proposed topology\-aligned contrastive loss\.

### 5\.2Theoretic\-Aligned Objective: Contrastive Attention as Phase Catalyst

Guided by our mechanistic framework, we construe the training objective not merely as likelihood maximization, but as a geometric intervention to optimize the alignment ratioρ​\(t\)\\rho\(t\)\(Eq\.[4](https://arxiv.org/html/2608.26139#S2.E4)\)\. To mitigate the gradient starvation \(Prop\.[2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)\) caused by syntactic noise, we introduce a topological auxiliary objective\.

While the standard language modeling lossℒL​M\\mathcal\{L\}\_\{LM\}provides a diffuse signal, we explicitly incentivize the emergence of the semantic operatorMs​e​m=μu​s​e​μs​r​cTM\_\{sem\}=\\mu\_\{use\}\\mu\_\{src\}^\{T\}\. Letℋt​g​t\\mathcal\{H\}\_\{tgt\}be the set of semantic heads, and𝒩\\mathcal\{N\}be a sampled set of syntactic distractor indices\. We formulate the contrastive objective as optimizing the margin between the source scoress​r​cs\_\{src\}and the average distractor scores¯n​e​g=𝔼n∈𝒩​\[sn\]\\bar\{s\}\_\{neg\}=\\mathbb\{E\}\_\{n\\in\\mathcal\{N\}\}\[s\_\{n\}\]:

ℒt​o​t​a​l=ℒL​M\+λ​∑h∈ℋt​g​tmax⁡\(0,η−\(ss​r​c\(h\)−s¯n​e​g\(h\)\)\)\\mathcal\{L\}\_\{total\}=\\mathcal\{L\}\_\{LM\}\+\\lambda\\sum\_\{h\\in\\mathcal\{H\}\_\{tgt\}\}\\max\\left\(0,\\eta\-\(s\_\{src\}^\{\(h\)\}\-\\bar\{s\}\_\{neg\}^\{\(h\)\}\)\\right\)\(8\)This formulation acts as a catalyst for the phase transition\. Considering the gradient with respect to the interaction matrixWQ​KW\_\{QK\}for an active constraint \(i\.e\., where the margin is violated\), the update direction acquires a specific rank\-1 component:

∇WQ​Kℒt​o​t​a​l\\displaystyle\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}\_\{total\}=∇WQ​KℒL​M\\displaystyle=\\nabla\_\{W\_\{QK\}\}\\mathcal\{L\}\_\{LM\}\(9\)−λ⋅𝕀a​c​t​i​v​e⋅μu​s​e​\(μs​r​c−μ¯n​e​g\)T⏟Steering Term\\displaystyle\\quad\-\\lambda\\cdot\\mathbb\{I\}\_\{active\}\\cdot\\underbrace\{\\mu\_\{use\}\(\\mu\_\{src\}\-\\bar\{\\mu\}\_\{neg\}\)^\{T\}\}\_\{\\text\{Steering Term\}\}where𝕀a​c​t​i​v​e\\mathbb\{I\}\_\{active\}is the indicator function for margin violation, andμ¯n​e​g\\bar\{\\mu\}\_\{neg\}is the centroid of the distractor embeddings\. Unlike the opaque∇ℒL​M\\nabla\\mathcal\{L\}\_\{LM\}, this steering term explicitly aligns the weight update with the semantic edgeμu​s​e​μs​r​cT\\mu\_\{use\}\\mu\_\{src\}^\{T\}while orthogonalizing it against syntactic directionsμu​s​e​μ¯n​e​gT\\mu\_\{use\}\\bar\{\\mu\}\_\{neg\}^\{T\}\. This mechanism effectively damps the curvature of the syntactic subspace relative to the semantic signal, thereby accelerating the traversal of the instability region \(Phase II\) and facilitating a more robust convergence to the systematic reasoning regime\.

### 5\.3Experiment Results

##### Universality and Structural Sparsity\.

To validate our framework’s generality, we profiled attention dynamics across architectures \(Llama\-3\.1\-8B, Qwen2\.5\-Coder\-7B\) and languages \(JavaScript, Python\)\. We define the semantic binding strengthSl,hS\_\{l,h\}for head\(l,h\)\(l,h\)as the expected attention mass from a usage tokenuuto its definition set𝒟​\(u\)\\mathcal\{D\}\(u\):Sl,h=𝔼u​\[∑v∈𝒟​\(u\)Au,v\(l,h\)\]\.S\_\{l,h\}=\\mathbb\{E\}\_\{u\}\[\\sum\_\{v\\in\\mathcal\{D\}\(u\)\}A\_\{u,v\}^\{\(l,h\)\}\]\.As visualized in Figure[8](https://arxiv.org/html/2608.26139#S5.F8), the results empirically confirm the Structural Sparsity predicted in Corollary[2\.8](https://arxiv.org/html/2608.26139#S2.Thmtheorem8): binding capabilities are concentrated in sparse, specialized heads rather than being diffusely distributed\. This pattern remains invariant across models and syntaxes, as further evidenced by Qwen2\.5\-Coder\-7B \(see App\.[A\.5](https://arxiv.org/html/2608.26139#A1.SS5)\)\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x8.png)

![Refer to caption](https://arxiv.org/html/2608.26139v1/x9.png)

Figure 8:Existence and Universality of Semantic Binding\.Attention heatmaps \(Sl,hS\_\{l,h\}\) for Llama\-3\.1\-8B on\(Left\)JavaScript and\(Right\)Python\. The consistent emergence of these specific heads across syntactically diverse languages supports the existence of shared language\-agnostic binding patterns across programming languages\.
##### Causal Verification via Head Ablation\.

To confirm the functional role of these heads, we performed an ablation on theVariable Cloze Completiontask by masking the Top\-8 binding heads against a random\-8 control\. As shown in Table[1](https://arxiv.org/html/2608.26139#S5.T1), while random masking caused a minor performance decline, ablating the binding heads led to a substantially larger drop\. This significant gap suggests that these specific circuits play an important functional role in reference resolution\.

##### Contrastive Tuning and Results\.

We fine\-tuned the model using the osed alignment objectiveℒt​o​t​a​l\\mathcal\{L\}\_\{total\}with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib62)\), applying the contrastive steering term exclusively to the Top\-8 Binding Heads identified through profiling\. This targeted intervention is designed to amplify the model’s naturally emerging binding circuits, rather than imposing semantic structure on arbitrary components\. As shown in Table[1](https://arxiv.org/html/2608.26139#S5.T1), this geometrically aligned strategy yields a substantially larger performance gain than pure cross\-entropy training: the improvement achieved by our method is more than twice the gain obtained from cross\-entropy alone, suggesting that explicitly sharpening attention geometry can be more sample\-efficient than implicit likelihood maximization in deep dependency learning tasks\. All detailed experiment methods and settings can be seen in App\.[B\.3](https://arxiv.org/html/2608.26139#A2.SS3)\.

##### Cross\-Language Generalization\.

Finally, to test whether our method captures fundamental binding logic rather than surface\-level syntax, we evaluate cross\-lingual transfer\. We apply the Python\-tuned model to JavaScript and the JavaScript\-tuned model to Python, without any additional fine\-tuning\. In both directions, our method consistently outperforms the baseline \(see App\.[A\.6](https://arxiv.org/html/2608.26139#A1.SS6)\)\. This cross\-lingual generalization indicates that the Semantic Binding Circuits reinforced by our objective are abstract and language\-agnostic, reflecting the shared topology of variable reference rather than language\-specific syntax\.

### 5\.4Generalization to Natural Language

To evaluate whether the proposed mechanism extends beyond code, we further study natural\-language deep dependency learning through pronoun and coreference resolution\. We map the notion of semantic dependency in code \(i\.e\., USE–DEF relations\) to antecedent–reference relations in natural language, where resolving a pronoun requires tracking latent relational structure across context\.

Following this formulation, we construct training pairs using AllenNLP\-based coreference extraction\(Gardneret al\.,[2018](https://arxiv.org/html/2608.26139#bib.bib63); Leeet al\.,[2018](https://arxiv.org/html/2608.26139#bib.bib64)\)\. Given a reference mention \(e\.g\., “his home”\), we identify its antecedent span \(e\.g\., “the house”\) and apply the same topology\-aligned contrastive objective used in the code setting\. To account for potential extraction noise in natural language, we additionally weight each pair by the confidence score returned by the coreference model\.

Table 2:Generalization to Natural Language\.We evaluate whether the proposed topology\-aligned objective extends beyond code by fine\-tuning on natural\-language coreference supervision and evaluating on WinoGrande\.We evaluate on the WinoGrande benchmark\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2608.26139#bib.bib65)\), a widely used benchmark for pronoun resolution and deep contextual dependency tracking\. Results are shown in Table[2](https://arxiv.org/html/2608.26139#S5.T2)\. Across both Qwen2\.5\-7B and Llama3\-8B, our method consistently improves over standard supervised fine\-tuning, suggesting that the proposed optimization mechanism generalizes beyond code\-specific structure\.

## 6Related Works

##### Attention Mechanisms\.

Existing research characterizes attention via mechanistic interpretability\(Clarket al\.,[2019](https://arxiv.org/html/2608.26139#bib.bib25); Liet al\.,[2020](https://arxiv.org/html/2608.26139#bib.bib26); Jawaharet al\.,[2019](https://arxiv.org/html/2608.26139#bib.bib27); Olssonet al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib9); Nandaet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib10); Conmyet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib11); Menget al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib12); Gevaet al\.,[2021](https://arxiv.org/html/2608.26139#bib.bib13); Merulloet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib14); Raiet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib15)\), training dynamics\(Tianet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib16); Maet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib17); Dherinet al\.,[2025](https://arxiv.org/html/2608.26139#bib.bib18); Zhanget al\.,[2025](https://arxiv.org/html/2608.26139#bib.bib19); Teehanet al\.,[2022](https://arxiv.org/html/2608.26139#bib.bib20); Havivet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib21); Geshkovskiet al\.,[2023](https://arxiv.org/html/2608.26139#bib.bib22); Kobayashiet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib23)\), and theoretical connections to corpus statistics\(Imet al\.,[2026](https://arxiv.org/html/2608.26139#bib.bib24); Daiet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib28); Prakashet al\.,[2024](https://arxiv.org/html/2608.26139#bib.bib29)\)\. Unlike these, we specifically address the unspecified conditions for realizing content\-independent relational operators and reliable pointer\-like binding\.

## 7Conclusion

In this work, we proposed a mechanistic framework for semantic emergence, characterizing it as a spectral competition where high\-curvature syntax initially masks deep dependency learning via gradient starvation\. Our experiments, spanning from few\-layer toy models trained on synthetic data to production\-scale LLMs, support the hypothesis that deep dependency learning can exhibit a sharp phase transition associated with the subspace alignment ratio\. Crucially, we demonstrated that this process can be influenced through targeted interventions: methods such as CoT supervision and our topology\-aligned contrastive objective help alleviate the syntactic barrier by rectifying gradient geometry\. These findings motivate further exploration of pre\-training strategies that move beyond simple likelihood maximization toward data curation and objectives that more explicitly encourage deep relational structure beyond surface statistical regularities\.

## Acknowledgments

This work was supported by the State Key Lab\. for Novel Software Technology \(KFKT2024B06\)\.

## Impact Statement

This paper presents work whose primary goal is to advance the field of Machine Learning by bridging the gap between mechanistic interpretability and model steering\. By establishing a theoretical link between gradient dynamics and the emergence of reasoning circuits, we offer a framework for more transparent and controllable pre\-training\.

However, practical application involves trade\-offs\. First, due to circuit polysemanticity, attention heads in LLMs often encode multiple entangled features; aggressive fine\-tuning of specific binding heads implies a risk of interfering with other latent capabilities\. Second, our topology\-aligned objective introduces computational overhead, as constraining internal activations significantly increases memory consumption \(VRAM\) during training compared to standard objectives\. Finally, our experiments on LLMs are concentrated in the code domain\. We utilize code because it offers explicit ground\-truth for semantic dependencies \(via ASTs\), whereas real\-world natural language lacks such unambiguous topological annotations\. Future work is required to develop robust probing and steering methods that can handle the linguistic ambiguity inherent in human communication\.

## References

- K\. Ahuja, V\. Balachandran, M\. Panwar, T\. He, N\. A\. Smith, N\. Goyal, and Y\. Tsvetkov \(2025\)Learning syntax without planting trees: understanding hierarchical generalization in transformers\.Transactions of the Association for Computational Linguistics13,pp\. 121–141\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- M\. Allamanis, M\. Brockschmidt, and M\. Khademi \(2017\)Learning to represent programs with graphs\.arXiv preprint arXiv:1711\.00740\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p5.1)\.
- M\. Alleman, J\. Mamou, M\. A\. Del Rio, H\. Tang, Y\. Kim, and S\. Chung \(2021\)Syntactic perturbations reveal representational correlates of hierarchical phrase structure in pretrained language models\.arXiv preprint arXiv:2104\.07578\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- J\. Bai, Y\. Wang, Y\. Chen, Y\. Yang, J\. Bai, J\. Yu, and Y\. Tong \(2021\)Syntax\-bert: improving pre\-trained transformers with syntax trees\.arXiv preprint arXiv:2103\.04350\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston \(2009\)Curriculum learning\.InProceedings of the 26th annual international conference on machine learning,pp\. 41–48\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.26139#S5.SS1.SSS3.Px1.p1.1)\.
- S\. Biderman, K\. Bicheno, and L\. Gao \(2022\)Datasheet for the pile\.arXiv preprint arXiv:2201\.07311\.Cited by:[§4](https://arxiv.org/html/2608.26139#S4.p1.1)\.
- S\. Biderman, U\. S\. Prashanth, L\. Sutawika, H\. Schoelkopf, Q\. Anthony, S\. Purohit, and E\. Raff \(2023a\)Emergent and predictable memorization in large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2608.26139#S4.p1.1)\.
- S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023b\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational Conference on Machine Learning,pp\. 2397–2430\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p5.1),[§4](https://arxiv.org/html/2608.26139#S4.p1.1)\.
- E\. Boix\-Adsera, O\. Saremi, E\. Abbe, S\. Bengio, E\. Littwin, and J\. Susskind \(2023\)When can transformers reason with abstract symbols?\.arXiv preprint arXiv:2310\.09753\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p2.1)\.
- C\. Caucheteux, A\. Gramfort, and J\. King \(2021\)Disentangling syntax and semantics in the brain with deep networks\.InInternational conference on machine learning,pp\. 1336–1348\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- M\. Chen \(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p5.1)\.
- K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. Manning \(2019\)What does bert look at? an analysis of bert’s attention\.arXiv preprint arXiv:1906\.04341\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Conmy, A\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-Alonso \(2023\)Towards automated circuit discovery for mechanistic interpretability\.Advances in Neural Information Processing Systems36,pp\. 16318–16352\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- Q\. Dai, B\. Heinzerling, and K\. Inui \(2024\)Representational analysis of binding in language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17468–17493\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- X\. Davies, M\. Nadeau, N\. Prakash, T\. R\. Shaham, and D\. Bau \(2023\)Discovering variable binding circuitry with desiderata\.arXiv preprint arXiv:2307\.03637\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- S\. R\. Dhayalkar \(2025\)Attention as binding: a vector\-symbolic perspective on transformer reasoning\.arXiv preprint arXiv:2512\.14709\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p2.1)\.
- B\. Dherin, M\. Munn, H\. Mazzawi, M\. Wunder, and J\. Gonzalvo \(2025\)Learning without training: the implicit dynamics of in\-context learning\.arXiv preprint arXiv:2507\.16003\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p5.1),[§5\.1\.1](https://arxiv.org/html/2608.26139#S5.SS1.SSS1.p1.1)\.
- G\. Felhi, J\. Le Roux, and D\. Seddah \(2022\)Exploiting inductive bias in transformers for unsupervised disentanglement of syntax and semantics with vaes\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5763–5776\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- J\. A\. Fodor and Z\. W\. Pylyshyn \(1988\)Connectionism and cognitive architecture: a critical analysis\.Cognition28\(1\-2\),pp\. 3–71\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima,et al\.\(2020\)The pile: an 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§4](https://arxiv.org/html/2608.26139#S4.p1.1)\.
- M\. Gardner, J\. Grus, M\. Neumann, O\. Tafjord, P\. Dasigi, N\. F\. Liu, M\. E\. Peters, M\. Schmitz, and L\. Zettlemoyer \(2018\)Allennlp: a deep semantic natural language processing platform\.InProceedings of workshop for NLP open source software \(NLP\-OSS\),pp\. 1–6\.Cited by:[§5\.4](https://arxiv.org/html/2608.26139#S5.SS4.p2.1)\.
- B\. Geshkovski, C\. Letrouit, Y\. Polyanskiy, and P\. Rigollet \(2023\)The emergence of clusters in self\-attention dynamics\.Advances in Neural Information Processing Systems36,pp\. 57026–57037\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 5484–5495\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- Z\. Gong, C\. Gao, Y\. Wang, W\. Gu, Y\. Peng, and Z\. Xu \(2022\)Source code summarization with structural relative position guided transformer\.In2022 IEEE International Conference on Software Analysis, Evolution and Reengineering \(SANER\),pp\. 13–24\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- B\. Gunel, J\. Du, A\. Conneau, and V\. Stoyanov \(2020\)Supervised contrastive learning for pre\-trained language model fine\-tuning\.arXiv preprint arXiv:2011\.01403\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.26139#S5.SS1.SSS3.Px3.p1.1)\.
- A\. Haviv, I\. Cohen, J\. Gidron, R\. Schuster, Y\. Goldberg, and M\. Geva \(2023\)Understanding transformer memorization recall through idioms\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 248–264\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§5\.3](https://arxiv.org/html/2608.26139#S5.SS3.SSS0.Px3.p1.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Dang,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p5.1),[§5\.1\.1](https://arxiv.org/html/2608.26139#S5.SS1.SSS1.p1.1)\.
- S\. Im, C\. Oh, Z\. Fang, and S\. Li \(2026\)How do transformers learn to associate tokens: gradient leading terms bring mechanistic interpretability\.arXiv preprint arXiv:2601\.19208\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Jawahar, B\. Sagot, and D\. Seddah \(2019\)What does bert learn about the structure of language?\.InACL 2019\-57th Annual Meeting of the Association for Computational Linguistics,Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Kalyanpur, O\. Biran, T\. Breloff, J\. Chu\-Carroll, A\. Diertani, O\. Rambow, and M\. Sammons \(2020\)Open\-domain frame semantic parsing using transformers\.arXiv preprint arXiv:2010\.10998\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- S\. Kobayashi, Y\. Akram, and J\. Von Oswald \(2024\)Weight decay induces low\-rank attention layers\.Advances in Neural Information Processing Systems37,pp\. 4481–4510\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- P\. Langley \(2000\)Crafting papers on machine learning\.InProceedings of the 17th International Conference on Machine Learning \(ICML 2000\),P\. Langley \(Ed\.\),Stanford, CA,pp\. 1207–1216\.Cited by:[§C\.6](https://arxiv.org/html/2608.26139#A3.SS6.SSS0.Px3.p2.1)\.
- K\. Lee, L\. He, and L\. Zettlemoyer \(2018\)Higher\-order coreference resolution with coarse\-to\-fine inference\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 687–692\.Cited by:[§5\.4](https://arxiv.org/html/2608.26139#S5.SS4.p2.1)\.
- L\. H\. Li, M\. Yatskar, D\. Yin, C\. Hsieh, and K\. Chang \(2020\)What does bert with vision look at?\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 5265–5275\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Ma, C\. Lin, D\. Lim, A\. Romero\-Soriano, P\. K\. Dokania, M\. Coates, P\. Torr, and S\. Lim \(2023\)Graph inductive biases in transformers without message passing\.InInternational Conference on Machine Learning,pp\. 23321–23337\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- G\. F\. Marcus \(1998\)Rethinking eliminative connectionism\.Cognitive psychology37\(3\),pp\. 243–282\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in gpt\.Advances in neural information processing systems35,pp\. 17359–17372\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Merullo, C\. Eickhoff, and E\. Pavlick \(2023\)Circuit component reuse across tasks in transformer language models\.arXiv preprint arXiv:2310\.08744\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- N\. Nanda, L\. Chan, T\. Lieberum, J\. Smith, and J\. Steinhardt \(2023\)Progress measures for grokking via mechanistic interpretability\.arXiv preprint arXiv:2301\.05217\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen,et al\.\(2022\)In\-context learning and induction heads\.arXiv preprint arXiv:2209\.11895\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1),[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- M\. Pezeshki, O\. Kaba, Y\. Bengio, A\. C\. Courville, D\. Precup, and G\. Lajoie \(2021\)Gradient starvation: a learning proclivity in neural networks\.Advances in Neural Information Processing Systems34,pp\. 1256–1272\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p2.1)\.
- A\. Power, Y\. Burda, H\. Edwards, I\. Babuschkin, and V\. Misra \(2022\)Grokking: generalization beyond overfitting on small algorithmic datasets\.arXiv preprint arXiv:2201\.02177\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- N\. Prakash, T\. R\. Shaham, T\. Haklay, Y\. Belinkov, and D\. Bau \(2024\)Fine\-tuning enhances existing mechanisms: a case study on entity tracking\.InThe Twelfth International Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- D\. Rai, Y\. Zhou, S\. Feng, A\. Saparov, and Z\. Yao \(2024\)A practical review of mechanistic interpretability for transformer\-based language models\.arXiv preprint arXiv:2407\.02646\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- V\. Raychev, P\. Bielik, and M\. Vechev \(2016\)Probabilistic model for code with decision trees\.ACM SIGPLAN Notices51\(10\),pp\. 731–747\.Cited by:[§5\.1\.2](https://arxiv.org/html/2608.26139#S5.SS1.SSS2.p1.1)\.
- J\. Russin, J\. Jo, R\. C\. O’Reilly, and Y\. Bengio \(2019\)Compositional generalization in a deep seq2seq model by separating syntax and semantics\.arXiv preprint arXiv:1904\.09708\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§5\.4](https://arxiv.org/html/2608.26139#S5.SS4.p3.1)\.
- R\. Teehan, M\. Clinciu, O\. Serikov, E\. Szczechla, N\. Seelam, S\. Mirkin, and A\. Gokaslan \(2022\)Emergent structures and training dynamics in large language models\.InProceedings of BigScience Episode\# 5–Workshop on Challenges & Perspectives in Creating Large Language Models,pp\. 146–159\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Tian, Y\. Wang, B\. Chen, and S\. S\. Du \(2023\)Scan and snap: understanding training dynamics and token composition in 1\-layer transformer\.Advances in neural information processing systems36,pp\. 71911–71947\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- O\. van der Wal, P\. Lesci, M\. Müller\-Eberstein, N\. Saphra, H\. Schoelkopf, W\. Zuidema, and S\. Biderman \(2025\)PolyPythias: stability and outliers across fifty language model pre\-training runs\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2608.26139#S4.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler,et al\.\(2022a\)Emergent abilities of large language models\.arXiv preprint arXiv:2206\.07682\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022b\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p4.1)\.
- L\. Weissweiler, V\. Hofmann, A\. Köksal, and H\. Schütze \(2022\)The better your syntax, the better your semantics? probing pretrained language models for the english comparative correlative\.arXiv preprint arXiv:2210\.13181\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.
- Y\. Wu, A\. Geiger, and R\. Millière \(2025\)How do transformers learn variable binding in symbolic programs?\.arXiv preprint arXiv:2505\.20896\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p1.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Fan \(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p5.1),[§5\.1\.1](https://arxiv.org/html/2608.26139#S5.SS1.SSS1.p1.1)\.
- Y\. Zhang, A\. K\. Singh, P\. E\. Latham, and A\. Saxe \(2025\)Training dynamics of in\-context learning in linear attention\.arXiv preprint arXiv:2501\.16265\.Cited by:[§6](https://arxiv.org/html/2608.26139#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Zhao, J\. Lin, Z\. Zhang, X\. Ren, Q\. Su, and X\. Sun \(2019\)Explicit sparse transformer: concentrated attention through explicit selection\.arXiv preprint arXiv:1912\.11637\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.26139#S5.SS1.SSS3.Px4.p1.1)\.
- S\. Zheng, S\. Jayasumana, B\. Romera\-Paredes, V\. Vineet, Z\. Su, D\. Du, C\. Huang, and P\. H\. Torr \(2015\)Conditional random fields as recurrent neural networks\.InProceedings of the IEEE international conference on computer vision,pp\. 1529–1537\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.26139#S5.SS1.SSS3.Px2.p1.1)\.
- J\. Zhou, Z\. Li, and H\. Zhao \(2020\)Parsing all: syntax and semantics, dependencies and spans\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 4438–4449\.Cited by:[§1](https://arxiv.org/html/2608.26139#S1.p3.1)\.

## Appendix AMore Experiment Results

All experiments in this section are performed with Qwen2\.5\-Coder\-7B\.

### A\.1Experimental Verification of Spectral Disparity

To validate the Spectral Disparity Assumption \([2\.1](https://arxiv.org/html/2608.26139#S2.Thmtheorem1)\), we probe the local geometry of the loss landscape usingIso\-Energy Perturbations\. We hypothesize that the Hessian eigenspectrum is highly anisotropic, with syntactic directions corresponding to large eigenvalues \(high curvature\) and semantic directions corresponding to small eigenvalues \(low curvature\)\.

##### Methodology\.

We rely on the local quadratic approximation of the loss landscape\. Around a local minimum, the change in loss under a perturbationδ\\deltais dominated by the HessianHH:Δ​ℒ≈12​δT​H​δ\\Delta\\mathcal\{L\}\\approx\\frac\{1\}\{2\}\\delta^\{T\}H\\delta\. To compare the spectral properties \(curvature\) of different subspaces, we apply perturbationsδ=α⋅v\\delta=\\alpha\\cdot valong directionsvvsampled strictly from the syntactic subspace \(𝒰s​y​n\\mathcal\{U\}\_\{syn\}\) or the semantic subspace \(𝒰s​e​m\\mathcal\{U\}\_\{sem\}\)\. Crucially, we enforcenormalization\(‖vs​y​n‖=‖vs​e​m‖=1\\\|v\_\{syn\}\\\|=\\\|v\_\{sem\}\\\|=1\)\. Under this constraint, the loss changeΔ​ℒ​\(α\)\\Delta\\mathcal\{L\}\(\\alpha\)becomes a direct proxy for the Rayleigh quotientvT​H​vv^\{T\}Hv, which measures the curvature along directionvv\. Steeper parabolic growth implies larger Hessian eigenvalues\.

##### Results: Empirical Validation of Assumption[2\.1](https://arxiv.org/html/2608.26139#S2.Thmtheorem1)\.

The experimental results are visualized in Figure[9](https://arxiv.org/html/2608.26139#A1.F9)\. The plot reveals a stark contrast in the local geometry of the two subspaces:

- •Syntactic Curvature \(Highλs​y​n\\lambda\_\{syn\}\):The loss landscape along syntactic directions \(red curve\) exhibits a sharp, steep valley\. Quantitatively, a small perturbation of magnitude\|α\|=0\.2\|\\alpha\|=0\.2results in a substantial loss increase ofΔ​ℒ≈1\.4\\Delta\\mathcal\{L\}\\approx 1\.4\. This rapid growth indicates that the HessianHHpossesses large eigenvalues along these eigenvectors, confirming the high curvature of𝒰s​y​n\\mathcal\{U\}\_\{syn\}\.
- •Semantic Curvature \(Lowλs​e​m\\lambda\_\{sem\}\):In contrast, the landscape along semantic directions \(blue curve\) is remarkably flat, forming a “plateau\.” The loss increase is negligible \(Δ​ℒ<0\.2\\Delta\\mathcal\{L\}<0\.2\) within the same perturbation range\. This empirically confirms that the semantic subspace is associated with near\-zero eigenvalues \(λs​e​m≈0\\lambda\_\{sem\}\\approx 0\)\.

Conclusion\.The experiment yields a curvature ratioλs​y​n/λs​e​m≫1\\lambda\_\{syn\}/\\lambda\_\{sem\}\\gg 1, providing direct physical evidence for theSpectral Disparitypostulate in Assumption[2\.1](https://arxiv.org/html/2608.26139#S2.Thmtheorem1)\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x10.png)

Figure 9:Loss landscape cross\-section\. The red curve \(Syntax\) shows significantly higher curvature \(steeper valley\) than the blue curve \(Semantics\), indicatingλs​y​n≫λs​e​m\\lambda\_\{syn\}\\gg\\lambda\_\{sem\}\.

### A\.2Validation of Low\-Rank Topological Operators

We experimentally test the core prediction of Prop\.[2\.7](https://arxiv.org/html/2608.26139#S2.Thmtheorem7): that semantic binding circuits collapse into rank\-1 topological operators \(Ms​e​m≈μu​s​e​μs​r​cTM\_\{sem\}\\approx\\mu\_\{use\}\\mu\_\{src\}^\{T\}\), whereas syntactic processing remains high\-rank\.

##### Methodology\.

We analyze the singular value spectrum of the interaction matricesWQ​K\(h\)∈ℝd×dW\_\{QK\}^\{\(h\)\}\\in\\mathbb\{R\}^\{d\\times d\}for all attention heads in Qwen2\.5\-Coder\-7B\. We utilize theStable Rank\(Effective Rank\) to quantify the spectral concentration of the spectrum:

re​f​f​\(WQ​K\)=‖σ‖22‖σ‖∞2=∑kσk2σmax2r\_\{eff\}\(W\_\{QK\}\)=\\frac\{\\\|\\sigma\\\|\_\{2\}^\{2\}\}\{\\\|\\sigma\\\|\_\{\\infty\}^\{2\}\}=\\frac\{\\sum\_\{k\}\\sigma\_\{k\}^\{2\}\}\{\\sigma\_\{\\max\}^\{2\}\}A valuere​f​f→1r\_\{eff\}\\to 1implies the matrix is dominated by a single principal component \(Rank\-1\)\.

##### Results\.

The spectral analysis is visualized in Figure[10](https://arxiv.org/html/2608.26139#A1.F10)\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x11.png)

Figure 10:Spectral Analysis of Interaction Matrices\.Left:Heatmap of effective rank across all layers and heads\. The emergence of semantic operators is sparse; only specific heads \(dark blue\) collapse to the low\-rank regime\.Right:Singular value spectrum comparison\. The Semantic Operator exhibits extreme spectral collapse withre​f​f≈1\.19r\_\{eff\}\\approx 1\.19, implying it functions as a precise Rank\-1 pointer\. The Syntactic Operator retains a high\-dimensional mixing structure \(re​f​f≈62\.18r\_\{eff\}\\approx 62\.18\)\.The empirical evidence strongly supports Prop\.[2\.7](https://arxiv.org/html/2608.26139#S2.Thmtheorem7):

1. 1\.Rank Collapse \(Semantic\):As shown in the right panel \(Blue Curve\), the semantic head exhibits a single component dominated spectrum\. The first singular value dominates the energy, resulting in an effective rank ofre​f​f≈1\.19r\_\{eff\}\\approx 1\.19\. This confirms that the learned matrixWQ​KW\_\{QK\}has effectively converged to the rank\-1 outer productμu​s​e​μs​r​cT\\mu\_\{use\}\\mu\_\{src\}^\{T\}\.
2. 2\.High\-Dimensional Mixing \(Syntactic\):In contrast, the syntactic head \(Red Curve\) shows a heavy\-tailed spectrum withre​f​f≈62\.18r\_\{eff\}\\approx 62\.18, indicating diffuse information processing typical of distributed representations\.
3. 3\.Sparsity:The heatmap \(Left Panel\) validates Corollary 3\.9\. Low\-rank operators \(dark blue pixels\) are rare and localized, confirming that the binding mechanism is specialized to a sparse subset of heads\.

### A\.3Validation of Systematic Gradient Emergence

Assumption[2\.6](https://arxiv.org/html/2608.26139#S2.Thmtheorem6)posits that the error signalγ\\gammaeventually decouples from context noise and aligns with a systematic semantic direction\. To verify this denoising effect, we analyze the geometric coherence of gradient vectors across different layers of the model\.

##### Methodology\.

We collect gradient vectorsg=∇hℒg=\\nabla\_\{h\}\\mathcal\{L\}with respect to the hidden states for specific semantic targets \(e\.g\., variable assignment operations\) acrossN=50N=50diverse contexts\. We employ two metrics to quantify gradient coherence:

1. 1\.PCA Explained Variance \(Top\-1\):We compute the Principal Component Analysis of the gradient set\{gi\}i=1N\\\{g\_\{i\}\\\}\_\{i=1\}^\{N\}\. A high explained variance ratio \(EVR\) for the first component indicates that the gradients are essentially 1\-dimensional \(pointing in a single semantic direction\) rather than diffuse\.
2. 2\.Pairwise Cosine Similarity:We measure the average cosine alignment1N2​∑i≠jcos⁡\(gi,gj\)\\frac\{1\}\{N^\{2\}\}\\sum\_\{i\\neq j\}\\cos\(g\_\{i\},g\_\{j\}\)\. High similarity implies the learning signal is consistent across different contexts \(context\-independent\)\.

##### Results\.

Figure[11](https://arxiv.org/html/2608.26139#A1.F11)illustrates the evolution of gradient geometry across network depth\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x12.png)

Figure 11:Emergence of Systematic Gradient Direction\.We plot the geometric properties of the backpropagated error signal against layer depth\.purplePurple Line \(PCA EVR\):The explained variance of the first principal component rises from≈0\.32\\approx 0\.32in shallow layers to\>0\.60\>0\.60in deep layers\. This indicates adimensionality collapse: the error signal transitions from high\-dimensional noise to a rank\-1 systematic vector\.orangeOrange Line \(Cosine Similarity\):The mean pairwise alignment increases monotonically, confirming that the gradient direction becomes robust to context noise as depth increases\. This validates the noise\-orthogonality postulate in Assumption[2\.6](https://arxiv.org/html/2608.26139#S2.Thmtheorem6)\.The data confirms a phase transition in the error signal quality\. In early layers \(L<15L<15\), gradients are noisy and context\-dependent \(low coherence\)\. In deeper layers \(L\>20L\>20\), the gradients lock onto a single dominant direction\. This Spectral Concentration ensures that the weight updateΔ​W∝γ​xT\\Delta W\\propto\\gamma x^\{T\}accumulates constructively along the topological edge, enabling the emergence of the binding circuit\.

### A\.4Visualizing the Variable Binding Subspace

Our theory predicts that semantic binding occurs within a specialized low\-dimensional subspace where variable identities are disentangled from their values and contexts\. To visualize thisBinding Subspace, we extract hidden states corresponding to variable references and project them into 2D space\.

##### Methodology\.

We generateN=1000N=1000random symbolic programs involving 26 variables \(a​…​za\\dots z\)\. We extract the activation vectorsh∈ℝdm​o​d​e​lh\\in\\mathbb\{R\}^\{d\_\{model\}\}at the exact token positions where a variable is referenced\. To isolate the binding\-relevant geometry from the high\-dimensional residual stream, we apply a two\-step dimensionality reduction:

1. 1\.Supervised Filtering:We train a sparse linear probe \(L1\-regularized Logistic Regression\) to predict the variable identity fromhh\. We select the topK=30K=30dimensions with the highest coefficients, effectively restricting the representation to the binding\-relevant subspace\.
2. 2\.Manifold Projection:We apply UMAP \(Uniform Manifold Approximation and Projection\) on these 30 dimensions to visualize the local topology\.

##### Results\.

The projected manifold is shown in Figure[12](https://arxiv.org/html/2608.26139#A1.F12)\.

![Refer to caption](https://arxiv.org/html/2608.26139v1/x13.png)

Figure 12:The Geometry of Variable Binding\.We visualize the internal representation of 26 distinct variables \(a​…​za\\dots z\) at Layer 20 of Qwen2\.5\-7B\. Each point represents a single occurrence of a variable in a unique random context\. We observe clear, distinctsemantic clustersfor each variable \(e\.g\., the red cluster ’v’ at the top, the green cluster ’l’ at the bottom\)\. Despite the randomness of the surrounding code \(contexts\), the model maps all instances of the same variable to a tightly packed region in the binding subspace\. This geometric disentanglement confirms that the model has learned abstract “Variable Types” as robust topological objects, independent of surface\-level syntax\.The visualization confirms that the “semantic subspace”𝒰s​e​m\\mathcal\{U\}\_\{sem\}is not hypothetical but a concrete, extractable structure\. The distinct islands for each variable demonstrate that the model has successfully factored the factorP​\(v​a​r​i​a​b​l​e\|c​o​n​t​e​x​t\)P\(variable\|context\)into a context\-independent code, a prerequisite for the reliable pointer\-passing mechanisms described in Remark[2\.10](https://arxiv.org/html/2608.26139#S2.Thmtheorem10)\.

### A\.5Structural Sparsity in Qwen

![Refer to caption](https://arxiv.org/html/2608.26139v1/x14.png)

![Refer to caption](https://arxiv.org/html/2608.26139v1/x15.png)

Figure 13:Existence and Universality of Semantic Binding\.Attention heatmaps \(Sl,hS\_\{l,h\}\) for Qwen2\.5\-Coder\-7B on\(Left\)JavaScript and\(Right\)Python\.
### A\.6Cross\-Language Generalization

The results are reported in Table[3](https://arxiv.org/html/2608.26139#A1.T3)\.

Table 3:Cross\-language generalization performance\. Models are trained on one programming language and evaluated on the other without additional fine\-tuning\. Our method consistently outperforms both Base and SFT across models and directions\.

## Appendix BExperiment Settings

### B\.1Experiments in Sec\.[3](https://arxiv.org/html/2608.26139#S3)

We utilize a causal “NanoTransformer” architecture trained from scratch for all experiments\. The model is a standard decoder\-only Transformer with pre\-normalization \(LayerNorm applied before the self\-attention and feed\-forward blocks\), GeLU activation, and absolute positional embeddings\. The vocabulary size is fixed atV=128V=128, partitioned into syntactic tokens \(e\.g\.,LET,=,;,RETURN\), variable identifiers \(v0​…​v19v\_\{0\}\\dots v\_\{19\}\), and numerical values\.

All models were trained using the AdamW optimizer on a single NVIDIA 3080Ti GPU\. To precisely control the dynamics, we employed a fixed seed \(4242\) for the final reporting of results, particularly for the causal interventions in Experiment 3\. The specific hyperparameters for each experimental setting are detailed in Table[4](https://arxiv.org/html/2608.26139#A2.T4)\.

All runs in this experiment utilize the same initialization seed \(42\)\.

Table 4:Hyperparameter configurations for the three controlled experiments\. Note that Experiment 2 uses a reduced model capacity to temporally extend the phase transition for better visualization\.##### Experiment 1: Visualizing Gradient Starvation\.

The objective of this experiment is to monitor the L2 norms of gradients at the embedding layer during the initial learning phase\. The dataset generator producesPointer Bindingsequences involving variable assignment \(e\.g\.,LET v1 = 5;\) and retrieval \(RETURN v1\)\. We define two token groups for monitoring:𝒯s​y​n\\mathcal\{T\}\_\{syn\}comprising syntax tokens \(indices0​…​90\\dots 9\) and𝒯s​e​m\\mathcal\{T\}\_\{sem\}comprising variable identifiers \(indices10​…​2910\\dots 29\)\. We record the average gradient norm‖∇Eℒ‖2\\\|\\nabla\_\{E\}\\mathcal\{L\}\\\|\_\{2\}for these subgroups at every step to visualize the delay in semantic learning relative to syntactic structure\.

##### Experiment 2: Escaping Softmax Saturation \(Curriculum\)\.

To investigate the how early attention concentration on syntactic tokens suppresses semantic learning and subsequent release, we employ a curriculum learning setup\. The model is initially trained on a variable assignment \(copy\) task\. At stept=1500t=1500, the data generator switches to an arithmetic addition task \(RETURN v1 \+ v2\)\. To capture the phase transition mechanics, we use a smaller model capacity \(dm​o​d​e​l=64d\_\{model\}=64\) and a reduced learning rate \(8×10−48\\times 10^\{\-4\}\)\. We monitor the attention weightsAQ​KA\_\{QK\}of the last layer, specifically tracking the attention mass allocated to operand tokens versus syntactic anchors\. High\-frequency sampling is applied around the task switch \(steps 1475–1600\) to resolve the crossover dynamics\.

##### Experiment 3: Causal Validation via Intervention\.

This experiment validates the causal role of the alignment ratioρ​\(t\)\\rho\(t\)using aDouble Dissociationprotocol\. We compare a standard control run against two intervention conditions using a Moderate Noise Generator\. \(1\)Syntax\-Suppressed:We artificially mask the gradients of syntactic tokens at the embedding layer \(∇s​y​n←0\\nabla\_\{syn\}\\leftarrow 0\) during backpropagation\. This intervention removes the “noise” from the syntactic subspace, theoretically increasingρ​\(t\)\\rho\(t\)and accelerating learning\. \(2\)Variable\-Disturbed:We inject Gaussian noise into the gradients of variable tokens \(∇v​a​r←∇v​a​r\+𝒩​\(0,1\)\\nabla\_\{var\}\\leftarrow\\nabla\_\{var\}\+\\mathcal\{N\}\(0,1\)\)\. This perturbation disrupts the semantic alignment directionMs​e​mM\_\{sem\}, decreasingρ​\(t\)\\rho\(t\)and delaying the emergence of reasoning\.

### B\.2Experiments in Sec\.[4](https://arxiv.org/html/2608.26139#S4)

We detail the experimental setup for the three mechanistic analyses performed on the Pythia model family\. All experiments were conducted using the HuggingFacetransformerslibrary withGPTNeoXForCausalLMarchitectures\. The analysis spans a training trajectory from initialization to convergence, utilizing Pythia\-160m for gradient\-sensitive probes and Pythia\-1\.4b for structural attention analysis\.

##### Model Configuration and Sampling\.

To capture the distinct phases of emergence, we employ different checkpoint granularities\. For the alignment dynamics \(Exp 1\), we use a dense sampling strategy to capture the phase transition\. For the structural and CoT analyses \(Exp 2 & 3\), we utilize three pivotal checkpoints \(t∈\{1​k,20​k,143​k\}t\\in\\\{1\\text\{k\},20\\text\{k\},143\\text\{k\}\\\}\) representing the heuristic phase, the transition point, and convergence\. Table[5](https://arxiv.org/html/2608.26139#A2.T5)summarizes these configurations\.

Table 5:Model specifications and checkpoint granularity\. Exp 1 uses dense sampling to capture the phase transition curve, while Exp 2 and 3 focus on snapshot comparisons\.
##### Exp 1: Probing the Alignment Ratioρ​\(t\)\\rho\(t\)\.

To validate the subspace competition hypothesis, we quantify the geometric alignment between the learned attention mechanism and the ideal semantic operation\. We define the alignment ratioρ​\(t\)\\rho\(t\)as the normalized projection of the model’s query\-key interaction matrixWQ​K\(t\)W\_\{QK\}^\{\(t\)\}onto the ground\-truth semantic operatorMs​e​mM\_\{sem\}:

ρ​\(t\)=⟨WQ​K\(t\),Ms​e​m⟩F‖WQ​K\(t\)‖F​‖Ms​e​m‖F\\rho\(t\)=\\frac\{\\langle W\_\{QK\}^\{\(t\)\},M\_\{sem\}\\rangle\_\{F\}\}\{\\\|W\_\{QK\}^\{\(t\)\}\\\|\_\{F\}\\\|M\_\{sem\}\\\|\_\{F\}\}\(10\)The matrixMs​e​mM\_\{sem\}is constructed as a low\-rank operator encoding the ideal dependency structure of the synthetic grammar\. We compute this metric across dense checkpoints to visualize the non\-monotonic transition from gradient starvation \(whereρ≈0\\rho\\approx 0\) to semantic locking\.

##### Exp 2: Retrospective Head Matching\.

We track the evolution of pointer mechanisms by generating synthetic dependency tasks \(Table[6](https://arxiv.org/html/2608.26139#A2.T6)\)\. To isolate the relevant circuit, we employ a “Time Travel” protocol\. We first identify the “Process Head” at the final checkpoint \(Step 143k\) that maximizes attention mass on the correct antecedent tokens \(L∗,H∗L^\{\*\},H^\{\*\}\)\. We then load earlier checkpoints \(1k, 20k\) and extract the attention patterns from this specific head indices, ensuring we are tracing the evolution of a fixed structural component rather than dynamically selecting the strongest head at each step\.

Table 6:Procedural generation templates for mechanistic probes\. Random variable names \(e\.g\.,x9k\) are used in Exp 3 to prevent token frequency artifacts\.
##### Exp 3: Gradient Saliency and CoT Restoration\.

To verify the signal restoration hypothesis, we measure the gradient flow through the network using a Gradient Saliency Probe\. For each checkpointt∈\{1​k,20​k,143​k\}t\\in\\\{1\\text\{k\},20\\text\{k\},143\\text\{k\}\\\}, we compute the gradient of the correct target logit with respect to the input embeddingsEE\. The selectivity score is defined as the log\-ratio of the gradient norm at the causal sourcexs​r​cx\_\{src\}versus a distractorxd​i​s​tx\_\{dist\}\. We generateN=60N=60unique random graph instances for each depthd∈\{10,…,50\}d\\in\\\{10,\\dots,50\\\}\. In the Implicit setting, the model must attend directly to the distant root value \(distractor: interleaved chain value\); in the CoT setting, the model is prompted to identify the immediate parent \(distractor: root value\), testing its ability to utilize local gradient pathways\.

### B\.3Our Loss

To enforce the mechanistic alignment between the model’s latent topology and the code’s semantic structure, we propose a composite objective function\. This objective combines a syntax\-aware generation loss with a geometric constraint on the attention mechanism\.

##### Variable\-Weighted Causal Modeling\.

Standard language modeling treats all tokens equally, often allowing the model to minimize loss via trivial syntactic patterns \(e\.g\., punctuation\) rather than resolving long\-range dependencies\. To counteract this, we introduce a re\-weighted cross\-entropy lossℒLM\\mathcal\{L\}\_\{\\text\{LM\}\}\. We define a weight vector𝐰∈ℝT\\mathbf\{w\}\\in\\mathbb\{R\}^\{T\}for a sequence of lengthTT, wherewt=αw\_\{t\}=\\alphaifxtx\_\{t\}belongs to a variable identifier \(definition or usage\), andwt=1w\_\{t\}=1otherwise\. Based on our preliminary profiling, we setα=5\.0\\alpha=5\.0to force the gradient descent to prioritize variable binding accuracy:

ℒLM=−1∑wt​∑t=1Twt​log⁡P​\(xt∣x<t;θ\)\\mathcal\{L\}\_\{\\text\{LM\}\}=\-\\frac\{1\}\{\\sum w\_\{t\}\}\\sum\_\{t=1\}^\{T\}w\_\{t\}\\log P\(x\_\{t\}\\mid x\_\{<t\};\\theta\)\(11\)

##### Contrastive Attention Alignment\.

To explicitly construct the retrieval circuit, we impose a hinge loss on a subset of “target heads”ℋt​a​r​g​e​t\\mathcal\{H\}\_\{target\}\. These heads are selected based on their pre\-trained alignment scores \(topK=8K=8heads\)\. For a given variable usage tokenqu​s​eq\_\{use\}, we require the attention mechanism to prioritize the corresponding definition tokenkd​e​fk\_\{def\}over random negative tokenskn​e​gk\_\{neg\}\. The alignment lossℒAlign\\mathcal\{L\}\_\{\\text\{Align\}\}is defined as:

ℒAlign=1\|ℋ\|​∑h∈ℋ𝔼q​\[max⁡\(0,γ−\(Ah​\(qu​s​e,kd​e​f\)−Ah​\(qu​s​e,kn​e​g\)\)\)\]\\mathcal\{L\}\_\{\\text\{Align\}\}=\\frac\{1\}\{\|\\mathcal\{H\}\|\}\\sum\_\{h\\in\\mathcal\{H\}\}\\mathbb\{E\}\_\{q\}\\left\[\\max\\left\(0,\\gamma\-\\left\(A\_\{h\}\(q\_\{use\},k\_\{def\}\)\-A\_\{h\}\(q\_\{use\},k\_\{neg\}\)\\right\)\\right\)\\right\]\(12\)whereAh​\(⋅\)A\_\{h\}\(\\cdot\)represents the attention weight, and we set the marginγ=0\.2\\gamma=0\.2\. To prevent training instability, the weight of this termλ​\(t\)\\lambda\(t\)follows a linear warmup schedule, peaking atλm​a​x=0\.5\\lambda\_\{max\}=0\.5\.

##### Dynamic Role Inference\.

Since the raw C\# training corpora lack explicit semantic labeling for “Definition” versus “Usage,” we implement a heuristic inference logic during data loading\. For every unique identifier, we sort its occurrences by position; the first occurrence is labeled asDef, and all subsequent occurrences are labeled asUse\. This allows us to construct the\(qu​s​e,kd​e​f\)\(q\_\{use\},k\_\{def\}\)pairs required forℒAlign\\mathcal\{L\}\_\{\\text\{Align\}\}without requiring an external static analysis compiler\.

##### Implementation and Training Details\.

We fine\-tuneQwen2\.5\-Coder\-7Busing Low\-Rank Adaptation \(LoRA\) to minimize memory overhead\. We target all linear projection layers \(Wq,Wk,Wv,WoW\_\{q\},W\_\{k\},W\_\{v\},W\_\{o\}and FFN gates\) with rankr=16r=16andα=32\\alpha=32\. The model is trained with a batch size of 1 and gradient accumulation steps of 32 \(effective batch size 32\)\. We use the AdamW optimizer with a learning rate of2×10−52\\times 10^\{\-5\}and a cosine decay schedule with 10% warmup\. The maximum sequence length is set to 2048 tokens to accommodate long\-context variable dependencies\.

## Appendix CDetailed Derivations and Theoretical Proofs

In this appendix, we provide the formal derivations and strict assumptions for the mechanistic claims made in Section[2](https://arxiv.org/html/2608.26139#S2)\.

### C\.1Formal Analysis of Gradient Starvation \(Supplementary to Sec\.[2\.2](https://arxiv.org/html/2608.26139#S2.SS2)\)

We rigorously justify Prop\.[2\.2](https://arxiv.org/html/2608.26139#S2.Thmtheorem2)by defining the residual distribution and applying spectral bounds\.

#### C\.1\.1Assumption on Residual Isotropy

To formally compare gradient projections, we assume the initial residuale=θ−θ∗e=\\theta\-\\theta^\{\*\}is isotropic up to the second moment\.

###### Assumption C\.1\(Second\-Order Isotropy\)\.

We assume the parameter residualeefollows a distribution𝒟e\\mathcal\{D\}\_\{e\}with zero mean and scaled identity covariance:

𝔼​\[e\]=0,𝔼​\[e​eT\]=σe2​I\\mathbb\{E\}\[e\]=0,\\quad\\mathbb\{E\}\[ee^\{T\}\]=\\sigma\_\{e\}^\{2\}I\(13\)Consequently, for any projection operatorPPonto a subspace of dimensionmm,𝔼​\[‖P​e‖2\]=σe2​m\\mathbb\{E\}\[\\\|Pe\\\|^\{2\}\]=\\sigma\_\{e\}^\{2\}m\.

#### C\.1\.2Proof of Starvation Ratio

Consider the quadratic approximationℒ​\(θ\)≈ℒ​\(θ∗\)\+12​eT​H​e\\mathcal\{L\}\(\\theta\)\\approx\\mathcal\{L\}\(\\theta^\{\*\}\)\+\\frac\{1\}\{2\}e^\{T\}He\. The gradient isg=H​eg=He\. We analyze the expected squared gradient norm in the syntactic subspace𝒰s​y​n\\mathcal\{U\}\_\{syn\}spanned by eigenvectors with eigenvalues\{λk\}k∈ℐs​y​n\\\{\\lambda\_\{k\}\\\}\_\{k\\in\\mathcal\{I\}\_\{syn\}\}\.

###### Proof\.

Using the eigendecompositionH=∑kλk​vk​vkTH=\\sum\_\{k\}\\lambda\_\{k\}v\_\{k\}v\_\{k\}^\{T\}:

𝔼​\[‖gs​y​n‖2\]=∑k∈ℐs​y​nλk2⋅𝔼​\[\(vkT​e\)2\]=σe2​∑k∈ℐs​y​nλk2\\mathbb\{E\}\[\\\|g\_\{syn\}\\\|^\{2\}\]=\\sum\_\{k\\in\\mathcal\{I\}\_\{syn\}\}\\lambda\_\{k\}^\{2\}\\cdot\\mathbb\{E\}\[\(v\_\{k\}^\{T\}e\)^\{2\}\]=\\sigma\_\{e\}^\{2\}\\sum\_\{k\\in\\mathcal\{I\}\_\{syn\}\}\\lambda\_\{k\}^\{2\}\(14\)Applying spectral bounds withλs​y​nmin=mink∈ℐs​y​n⁡λk\\lambda\_\{syn\}^\{\\min\}=\\min\_\{k\\in\\mathcal\{I\}\_\{syn\}\}\\lambda\_\{k\}:

𝔼​\[‖gs​y​n‖2\]≥ms​y​n​\(λs​y​nmin\)2​σe2\\mathbb\{E\}\[\\\|g\_\{syn\}\\\|^\{2\}\]\\geq m\_\{syn\}\(\\lambda\_\{syn\}^\{\\min\}\)^\{2\}\\sigma\_\{e\}^\{2\}\(15\)Assuming spectral disparityλs​y​nmin≫λs​e​mmax\\lambda\_\{syn\}^\{\\min\}\\gg\\lambda\_\{sem\}^\{\\max\}, the ratio of expected gradient norms satisfies:

𝔼​\[‖gs​y​n‖2\]𝔼​\[‖gs​e​m‖2\]≥ms​y​nms​e​m​\(λs​y​nminλs​e​mmax\)2≫1\\frac\{\\mathbb\{E\}\[\\\|g\_\{syn\}\\\|^\{2\}\]\}\{\\mathbb\{E\}\[\\\|g\_\{sem\}\\\|^\{2\}\]\}\\geq\\frac\{m\_\{syn\}\}\{m\_\{sem\}\}\\left\(\\frac\{\\lambda\_\{syn\}^\{\\min\}\}\{\\lambda\_\{sem\}^\{\\max\}\}\\right\)^\{2\}\\gg 1\(16\)This confirms that the gradient energy is dominated by the syntactic subspace\. ∎

### C\.2Derivation of the Vanishing Gradient Barrier \(Supplementary to Sec\.[2\.3](https://arxiv.org/html/2608.26139#S2.SS3)\)

We derive the exponential decay bound using the correct chain rule\.

#### C\.2\.1Derivative via Chain Rule

The gradient w\.r\.t\. a scoress​e​ms\_\{sem\}is given by the Jacobian of the softmaxA​\(s\)A\(s\):

∂ℒ∂ss​e​m=∑k∂ℒ∂Ak​∂Ak∂ss​e​m=As​e​m​\(∂ℒ∂As​e​m−∑kAk​∂ℒ∂Ak\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{sem\}\}=\\sum\_\{k\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}\\frac\{\\partial A\_\{k\}\}\{\\partial s\_\{sem\}\}=A\_\{sem\}\\left\(\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{sem\}\}\-\\sum\_\{k\}A\_\{k\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}\\right\)\(17\)

#### C\.2\.2Exponential Decay under Boundedness

###### Assumption C\.2\(Bounded Downstream Gradient\)\.

We assume the downstream gradient is bounded:\|∂ℒ∂Ak\|≤C\\left\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}\\right\|\\leq C\.

###### Proof\.

LetDk=∂ℒ∂AkD\_\{k\}=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\_\{k\}\}\. The term in parenthesis is bounded by2​C2C\. In the Top\-1 syntactic domination regime \(ss​y​n≫ss​e​ms\_\{syn\}\\gg s\_\{sem\}\), we haveAs​e​m≈e−\(ss​y​n−ss​e​m\)A\_\{sem\}\\approx e^\{\-\(s\_\{syn\}\-s\_\{sem\}\)\}\. Thus:

\|∂ℒ∂ss​e​m\|≤2​C⋅As​e​m=O​\(e−\(ss​y​n−ss​e​m\)\)\\left\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial s\_\{sem\}\}\\right\|\\leq 2C\\cdot A\_\{sem\}=O\\left\(e^\{\-\(s\_\{syn\}\-s\_\{sem\}\)\}\\right\)\(18\)This confirms the exponential vanishing of the semantic learning signal\. ∎

### C\.3Hebbian Reconstruction Analysis \(Supplementary to Sec\.[2\.4](https://arxiv.org/html/2608.26139#S2.SS4)\)

We rigorously justify the rank\-1 update direction by analyzing the sensitivity of the error signal to noise\.

#### C\.3\.1Teacher Graph and Error Signal Signs

We assume a generative model whereyj=f​\(xi∗​\(j\)\)y\_\{j\}=f\(x\_\{i^\{\*\}\(j\)\}\)andi∗​\(j\)i^\{\*\}\(j\)is the unique true dependency\. The parameter update is proportional to−∇Wℒ≈−∑j,iγj​i​xj​xiT\-\\nabla\_\{W\}\\mathcal\{L\}\\approx\-\\sum\_\{j,i\}\\gamma\_\{ji\}x\_\{j\}x\_\{i\}^\{T\}\. For the correct edge, increasing attention decreases loss, so𝔼​\[γj​i∗\]=−c\\mathbb\{E\}\[\\gamma\_\{ji^\{\*\}\}\]=\-cwithc\>0c\>0\. For irrelevant edges,𝔼​\[γj​i\]≈0\\mathbb\{E\}\[\\gamma\_\{ji\}\]\\approx 0\.

#### C\.3\.2Noise Sensitivity and Rank\-1 Dominance

We decompose embeddings asx=μ\+δx=\\mu\+\\delta, whereδ\\deltais zero\-mean noise with covarianceΣδ\\Sigma\_\{\\delta\}\. We assume the error signalγ​\(x\)\\gamma\(x\)is locally smooth w\.r\.t\. noise\.

###### Assumption C\.3\(Weak Noise Sensitivity\)\.

We approximateγ\\gammavia a first\-order Taylor expansion around the signalμ\\mu:

γ​\(μ\+δ\)≈γ​\(μ\)\+∇γ​\(μ\)T​δ\\gamma\(\\mu\+\\delta\)\\approx\\gamma\(\\mu\)\+\\nabla\\gamma\(\\mu\)^\{T\}\\delta\(19\)This implies that the correlation between error and noise is of second order:𝔼​\[γ​δ\]≈O​\(𝔼​\[‖δ‖2\]\)\\mathbb\{E\}\[\\gamma\\delta\]\\approx O\(\\mathbb\{E\}\[\\\|\\delta\\\|^\{2\}\]\)\.

Using this expansion, the expected update direction becomes:

𝔼​\[−∇Wℒ\]\\displaystyle\\mathbb\{E\}\[\-\\nabla\_\{W\}\\mathcal\{L\}\]=𝔼​\[∑j−\(γ​\(μj\)\+ϵ\)​\(μj\+δj\)​\(μi∗\+δi∗\)T\]\\displaystyle=\\mathbb\{E\}\\left\[\\sum\_\{j\}\-\(\\gamma\(\\mu\_\{j\}\)\+\\epsilon\)\(\\mu\_\{j\}\+\\delta\_\{j\}\)\(\\mu\_\{i^\{\*\}\}\+\\delta\_\{i^\{\*\}\}\)^\{T\}\\right\]\(20\)=∑jc⋅μj​μi∗T\+O​\(𝔼​\[‖δ‖2\]\)\\displaystyle=\\sum\_\{j\}c\\cdot\\mu\_\{j\}\\mu\_\{i^\{\*\}\}^\{T\}\+O\(\\mathbb\{E\}\[\\\|\\delta\\\|^\{2\}\]\)The cross\-terms \(e\.g\.,γ​μ​δT\\gamma\\mu\\delta^\{T\}\) vanish in expectation due to the independence ofδ\\deltaacross positions or its zero mean\. Thus, the update is dominated by the rank\-1 topological operatorμu​s​e​μs​r​cT\\mu\_\{use\}\\mu\_\{src\}^\{T\}, with noise terms acting as a second\-order perturbation\.

### C\.4Derivation of the Alignment Threshold \(Supplementary to Sec\.[2\.5](https://arxiv.org/html/2608.26139#S2.SS5)\)

We derive the stability condition forρ​\(t\)\\rho\(t\)\. Let the gradient be∇=S\+N\\nabla=S\+N, whereSSis the systematic semantic direction andNNis zero\-mean noise\. We explicitly assume the noise is uncorrelated with the signal in expectation:

𝔼​\[⟨S,N⟩\]=0\\mathbb\{E\}\[\\langle S,N\\rangle\]=0\(21\)The expected alignment ratio \(cosine similarity squared\) is:

𝔼​\[ρ\]≈‖S‖2𝔼​\[‖∇‖2\]=‖S‖2‖S‖2\+𝔼​\[‖N‖2\]\+2​𝔼​\[⟨S,N⟩\]=11\+SNR−1\\mathbb\{E\}\[\\rho\]\\approx\\frac\{\\\|S\\\|^\{2\}\}\{\\mathbb\{E\}\[\\\|\\nabla\\\|^\{2\}\]\}=\\frac\{\\\|S\\\|^\{2\}\}\{\\\|S\\\|^\{2\}\+\\mathbb\{E\}\[\\\|N\\\|^\{2\}\]\+2\\mathbb\{E\}\[\\langle S,N\\rangle\]\}=\\frac\{1\}\{1\+\\text\{SNR\}^\{\-1\}\}\(22\)whereSNR=‖S‖2/𝔼​\[‖N‖2\]\\text\{SNR\}=\\\|S\\\|^\{2\}/\\mathbb\{E\}\[\\\|N\\\|^\{2\}\]\. The thresholdτs​t​a​b​l​e\\tau\_\{stable\}corresponds to the critical SNR where the systematic signalSSdominates the variance ofNN\.

### C\.5Formal Argument for Pointer Priority \(Supplementary to Remark[2\.10](https://arxiv.org/html/2608.26139#S2.Thmtheorem10)\)

We formalize the precedence of Pointer \(i→ji\\to j\) over Value \(i→ki\\to k\) based on sample complexity and signal accumulation\.

#### C\.5\.1Mechanism vs\. Fact

- •Pointer \(Mechanism\): Relies on Type\-level consistency \(Nt​y​p​eN\_\{type\}samples\)\.
- •Value \(Fact\): Relies on Instance\-level consistency \(Ni​n​s​tN\_\{inst\}samples\)\.

ClearlyNt​y​p​e≫Ni​n​s​tN\_\{type\}\\gg N\_\{inst\}\.

#### C\.5\.2SNR Scaling Law

We model the learning process as signal accumulation under additive Gaussian noise\. Letggbe the per\-sample gradient\. The cumulative update overNNsamples isΔ​W∼∑n=1N\(gs​i​g​n​a​l\+ξn​o​i​s​e\)\\Delta W\\sim\\sum\_\{n=1\}^\{N\}\(g\_\{signal\}\+\\xi\_\{noise\}\)\.

- •Signal magnitude:‖∑gs​i​g​n​a​l‖∝N\\\|\\sum g\_\{signal\}\\\|\\propto N
- •Noise magnitude \(Random Walk\):‖∑ξn​o​i​s​e‖∝N\\\|\\sum\\xi\_\{noise\}\\\|\\propto\\sqrt\{N\}

The effective Signal\-to\-Noise Ratio for the learned circuit scales as:

SNR​\(N\)=SignalNoise∝NN=N\\text\{SNR\}\(N\)=\\frac\{\\text\{Signal\}\}\{\\text\{Noise\}\}\\propto\\frac\{N\}\{\\sqrt\{N\}\}=\\sqrt\{N\}\(23\)
Comparing the Pointer and Value circuits:

SNRp​t​rSNRv​a​l≈Nt​y​p​eNi​n​s​t≫1\\frac\{\\text\{SNR\}\_\{ptr\}\}\{\\text\{SNR\}\_\{val\}\}\\approx\\sqrt\{\\frac\{N\_\{type\}\}\{N\_\{inst\}\}\}\\gg 1\(24\)SinceSNRp​t​r\\text\{SNR\}\_\{ptr\}grows significantly faster,ρp​t​r​\(t\)\\rho\_\{ptr\}\(t\)crosses the stability thresholdτs​t​a​b​l​e\\tau\_\{stable\}earlier thanρv​a​l​\(t\)\\rho\_\{val\}\(t\), ensuring the pointer mechanism emerges first\.

### C\.6Accelerating Semantic Emergence via CoT

While CoT is widely used for complex reasoning, its mechanistic basis remains under\-explored\. We show that CoT mitigates the Softmax Saturation barrier \(Thm\.[2\.4](https://arxiv.org/html/2608.26139#S2.Thmtheorem4)\) by making intermediate states explicit\. At inference time, explicit traces expose latent states as context tokens, improving semantic retrieval\. During training, supervising these traces adds a local loss term that provides a parallel gradient pathway, alleviating credit assignment bottlenecks\.

##### Problem Formulation\.

Consider a compositional taskx→yx\\to ymediated by an intermediate sequencezz\(e\.g\.,zzis the resolved binding/value\-fetch result, andyyis the final output\)\.

- •Implicit Reasoning:The model outputsyydirectly, i\.e\.,P​\(y∣x\)P\(y\\mid x\), while the intermediatezzremains a latent computation\.
- •CoT Prompting \(Inference\-time\):The model generates the joint sequence\(z,y\)\(z,y\), i\.e\.,P​\(z,y∣x\)=P​\(z∣x\)​P​\(y∣x,z\)P\(z,y\\mid x\)=P\(z\\mid x\)\\,P\(y\\mid x,z\)\.

##### Inference\-time CoT: Externalization as State Injection\.

Implicit reasoning must resolve multi\-hop dependencies \(e\.g\.,x→z→yx\\to z\\to y\) within a single forward pass\. Under softmax saturation \(Thm\.[2\.4](https://arxiv.org/html/2608.26139#S2.Thmtheorem4)\), attention can be dominated by syntactic tokens, making semantic predecessor retrieval unreliable and degradingP​\(y∣x\)P\(y\\mid x\)\. CoT prompting externalizes the intermediate statezzas an explicit token, converting the hard conditionalP​\(y∣x\)P\(y\\mid x\)into two easier conditionalsP​\(z∣x\)P\(z\\mid x\)andP​\(y∣x,z\)P\(y\\mid x,z\), which stabilizes downstream semantic attention\.

##### Training\-time CoT Supervision: Objective\-level Injection\.

We now analyze the regime where intermediate traceszzare supervised during training, as in CoT supervised fine\-tuning or CoT distillation\. In this case, the training objective factorizes as:

ℒCoT​\-​sup\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{CoT\\text\{\-\}sup\}\}=−log⁡P​\(z,y∣x\)\\displaystyle=\-\\log P\(z,y\\mid x\)\(25\)=−log⁡P​\(z∣x\)−log⁡P​\(y∣x,z\)\\displaystyle=\-\\log P\(z\\mid x\)\-\\log P\(y\\mid x,z\)=ℒz\+ℒy∣z\.\\displaystyle=\\mathcal\{L\}\_\{z\}\+\\mathcal\{L\}\_\{y\\mid z\}\.Unlike inference\-time prompting, this objective introduces an explicit loss termℒz\\mathcal\{L\}\_\{z\}that directly supervises the intermediate statezz, creating a parallel and more local gradient signal for binding\-related parameters\.

###### Proposition C\.4\(Gradient Attenuation in Implicit Reasoning\)\.

In the implicit setting, the lossℒi​m​p=−log⁡P​\(y∣x\)\\mathcal\{L\}\_\{imp\}=\-\\log P\(y\\mid x\)depends on the binding\-circuit parametersθs​e​m\\theta\_\{sem\}only through the downstream computationy=g​\(z\)y=g\(z\)\. Letss​e​ms\_\{sem\}denote the binding\-relevant score \(e\.g\., an attention logit\)\. Using Jacobian notation \(denominator layout\), the chain rule gives:

b\\displaystyle b:=Jy​\(z\)⊤​∇yℒi​m​p,\\displaystyle=J\_\{y\}\(z\)^\{\\top\}\\,\\nabla\_\{y\}\\mathcal\{L\}\_\{imp\},\(26\)∇θb​i​n​dℒi​m​p\\displaystyle\\nabla\_\{\\theta\_\{bind\}\}\\mathcal\{L\}\_\{imp\}=Jsb​i​n​d​\(θb​i​n​d\)⊤​Jz​\(sb​i​n​d\)⊤​b\.\\displaystyle=J\_\{s\_\{bind\}\}\(\\theta\_\{bind\}\)^\{\\top\}\\,J\_\{z\}\(s\_\{bind\}\)^\{\\top\}\\,b\.whereJa​\(b\):=∂a∂bJ\_\{a\}\(b\):=\\frac\{\\partial a\}\{\\partial b\}\. This path is vulnerable to two failure modes:

1. 1\.Downstream Insensitivity:If the JacobianJy​\(z\)=∂y∂zJ\_\{y\}\(z\)=\\frac\{\\partial y\}\{\\partial z\}has small spectral norm, the signal can vanish before reachingθs​e​m\\theta\_\{sem\}\.
2. 2\.Upstream Gating:As shown in Lemma[2\.3](https://arxiv.org/html/2608.26139#S2.Thmtheorem3), the local derivativeJz​\(sb​i​n​d\)=∂z∂sb​i​n​dJ\_\{z\}\(s\_\{bind\}\)=\\frac\{\\partial z\}\{\\partial s\_\{bind\}\}is multiplicatively suppressed under softmax saturation\. In the syntactic phase, this term can effectively gate the entire gradient chain\.

###### Proposition C\.5\(Parallel Gradient Pathway under CoT Supervision\)\.

Under training\-time CoT supervision, the total gradient becomes an additive composition:

∇θb​i​n​dt​o​t​a​l=∇θb​i​n​dℒy∣z\+∇θb​i​n​dℒz⏟Injected Gradient\.\\nabla\_\{\\theta\_\{bind\}\}^\{total\}=\\nabla\_\{\\theta\_\{bind\}\}\\mathcal\{L\}\_\{y\\mid z\}\+\\underbrace\{\\nabla\_\{\\theta\_\{bind\}\}\\mathcal\{L\}\_\{z\}\}\_\{\\text\{Injected Gradient\}\}\.\(27\)The injected term∇θb​i​n​dℒz\\nabla\_\{\\theta\_\{bind\}\}\\mathcal\{L\}\_\{z\}depends only on predicting the intermediate statezz\. It provides a parallel learning signal that bypasses the downstream sensitivity termJy​\(z\)J\_\{y\}\(z\)and remains active even when the implicit pathway is weak or gated by saturation\. When‖∇θb​i​n​dℒy∣z‖\\\|\\nabla\_\{\\theta\_\{bind\}\}\\mathcal\{L\}\_\{y\\mid z\}\\\|is near\-zero due to starvation/saturation, this additional pathway*effectively short\-circuits*the optimization bottleneck\.

###### Corollary C\.6\(Semantic Alignment Boost under CoT Supervision\)\.

Recall the semantic operatorMs​e​mM\_\{sem\}defined in[Section2\.5](https://arxiv.org/html/2608.26139#S2.SS5)\. LetΔi​m​p\\Delta^\{imp\}denote the implicit gradient component fromℒy∣z\\mathcal\{L\}\_\{y\\mid z\}, and letΔi​n​j\\Delta^\{inj\}denote the injected gradient fromℒz\\mathcal\{L\}\_\{z\}\. Assuming the supervised traces are*valid*, i\.e\.,zzencodes dependency edges that causally support the correct outputyy, we have

⟨Δi​n​j,Ms​e​m⟩F≫0\.\\langle\\Delta^\{inj\},M\_\{sem\}\\rangle\_\{F\}\\gg 0\.\(28\)Therefore the total semantic projection increases additively:

⟨Δt​o​t​a​l,Ms​e​m⟩F=⟨Δi​m​p,Ms​e​m⟩F\+⟨Δi​n​j,Ms​e​m⟩F,\\langle\\Delta^\{total\},M\_\{sem\}\\rangle\_\{F\}=\\langle\\Delta^\{imp\},M\_\{sem\}\\rangle\_\{F\}\+\\langle\\Delta^\{inj\},M\_\{sem\}\\rangle\_\{F\},\(29\)This increases the semantic projection of the update direction and thus tends to raise the alignment ratioρ​\(t\)\\rho\(t\)in[Equation4](https://arxiv.org/html/2608.26139#S2.E4)\. As a result, CoT supervision accelerates the crossing of the stability thresholdτs​t​a​b​l​e\\tau\_\{stable\}\.