代码语言模型中的因果句法回路：预算与探针决定何种现象看起来具有局部性

# Causal Syntax Circuits in CodeLMs: Budget and Probe Decide What Looks Localised
Source: [https://arxiv.org/html/2610.20683](https://arxiv.org/html/2610.20683)
\\workshoptitle

Linguistic Principles for Foundation Models \(LP4FM\)

###### Abstract

Code models write Python when the prompt does not say which language to use\. We ask a mechanistic question about that habit: is it produced by a small, findable group of components inside the model? We run one pipeline—mean ablation, activation patching, greedy circuit extraction, a random\-control significance test, and activation steering—over 55 model–language settings spanning ten open code models \(five base and five instruction\-tuned\), three target languages \(Rust, C\#, PHP\), two prompt suites, and two component budgets\. We report five findings\. First, the preference is large, grows with model size, and survives instruction tuning: it is larger on every instruction\-tuned checkpoint than on its base version, by 12% on average\. Second, its measured size is mostly a property of the probe; replacing our aligned prompts with XLCoST parallel functions takes the same measurement on the same models from about1\.61\.6to about0\.010\.01\. Third, whether the preference looks localised depends on how many components the search may keep: with a budget of 12, 1 of 15 settings yields a circuit that beats size\-matched random controls after multiple\-comparison correction, and with a budget of 40, 6 of 15 do\. Fourth, interventions that a smaller\-budget study reports as uniformly failing do sometimes succeed: patching makes a Python prompt prefer target syntax in 2 of 45 settings, and steering changes the most likely next token in 9 of 45\. Fifth, every circuit whose effect is significant but points*away*from the target language comes from a setting where circuit extraction failed, which makes such results a diagnostic rather than a phenomenon\. The practical lesson is that a claim about whether a behaviour is localised can turn on analysis choices that papers rarely report\. We release the full grid so those choices can be varied\.

## 1Introduction

Ask a code model to “sort a list” and it will usually hand you Python\. Ask it in a Rust repository, with a Rust file open, and it will often still reach for Python idioms unless the prompt spells out the language\. Benchmarks report the same pattern at scale: base code models score higher on Python than on the other languages they were trained on, and they fall back to Python when the target is left open\([Cassano et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib4);[Zheng et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib28)\)\. We call this tendency*programming\-language defaultness*\. The two facts are separate, and we do not claim one causes the other; the most likely common cause is that Python is over\-represented in code training corpora\.

#### What behaviour would count as correct\.

We are not arguing that a Python default is a bug: when a task admits many languages, a model has to break the tie somehow\. The property we care about is narrower and testable—a user should be able to*move*the default—so our question is mechanistic, not normative\. Is the preference carried by a small group of components that an inference\-time edit could redirect, in the way that refusal\([Arditi et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib1)\)and natural\-language identity\([Tang et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib21);[Kojima et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib13)\)have been reported to be? A yes would make cheap, weight\-free language control possible for teams whose codebases are not Python\. A no tells practitioners to spend their effort elsewhere\.

#### Why revisit this\.

An earlier version of this study answered that question in the negative across 15 model–language settings: the preference was measurable everywhere, a specific circuit survived correction in one setting, and neither patching nor steering ever changed the generated language\. That conclusion rested on two choices that looked like implementation details—keeping the top 12 components after ablation ranking, and measuring on a hand\-built suite of aligned function pairs—plus a scope restriction to base models\.

#### Contributions\.

We extend the grid from 15 settings to 55 by adding \(i\) five instruction\-tuned checkpoints, \(ii\) a component budget of 40, matching the scale used by sparse\-feature circuit work\([Marks et al\., 2025](https://arxiv.org/html/2610.20683#bib.bib31)\), and \(iii\) a second prompt suite drawn from XLCoST parallel functions\([Zhu et al\., 2022](https://arxiv.org/html/2610.20683#bib.bib29)\)\.111Code, configuration files, prompt suites, and the result files for all 55 runs are available at[https://github\.com/Kitsunnneee/causal\-syntax\-circuits](https://github.com/Kitsunnneee/causal-syntax-circuits)\.Table[1](https://arxiv.org/html/2610.20683#S1.T1)is the whole result in one place, and each of the five findings below is stated once, in a box, where its evidence is presented\. Changing those three things changes the conclusion\.

Table 1:Every claim in the paper, under every probe: the number of settings supporting the claim out of the settings run\. Only the first row is stable\. The rest move when the component budget or the prompt suite changes, which is the paper’s main point\. Definitions are in §[3](https://arxiv.org/html/2610.20683#S3)\.

## 2Localising behaviour, and what it has meant for language identity

#### Localising behaviour by intervening\.

Our methods come from causal mediation analysis\([Vig et al\., 2020](https://arxiv.org/html/2610.20683#bib.bib25)\)and activation patching\([Meng et al\., 2022](https://arxiv.org/html/2610.20683#bib.bib16)\): a behaviour is traced by measuring how a metric moves when internal activations are removed or transplanted between runs\. Circuit\-level analyses in this tradition include the indirect\-object\-identification circuit in GPT\-2\([Wang et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib26)\)and automated circuit discovery\([Conmy et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib5);[Syed et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib20)\)\. We follow reported best practice for patching metrics\([Zhang and Nanda, 2024](https://arxiv.org/html/2610.20683#bib.bib27)\), and our greedy extraction approximates edge attribution patching\([Syed et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib20)\)at the node level\. Because such interventions can read structure into arbitrary substructure, causal\-abstraction work\([Geiger et al\., 2021](https://arxiv.org/html/2610.20683#bib.bib8);[Geiger et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib9)\)and interpretability surveys\([Bereska and Gavves, 2024](https://arxiv.org/html/2610.20683#bib.bib2)\)stress the need for controls, which is why every localisation claim here is gated on a comparison against size\-matched random component sets\.

#### Directions and steering\.

A parallel line holds that concepts are encoded as directions rather than as single neurons\([Elhage et al\., 2022](https://arxiv.org/html/2610.20683#bib.bib7);[Park et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib18)\), and that adding such a direction during the forward pass steers behaviour\([Turner et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib24);[Zou et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib30);[Li et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib14)\)\. Single directions reportedly mediate behaviours as specific as refusal\([Arditi et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib1)\), and small component sets can carry a task as a vector\([Todd et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib23);[Hendel et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib11)\)\. Our steering estimator is the standard difference\-in\-means construction; our contribution is to calibrate it\.

#### Language identity, natural and programming\.

The closest precedent is natural\-language identity: a small set of “language\-specific neurons”, concentrated in the first and last layers, can be toggled to change a multilingual model’s output language\([Tang et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib21);[Kojima et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib13)\)\. That literature ties output\-language preference to training\-data frequency, and Python defaultness is the code\-side analogue\. Programming languages differ in ways that matter for whether the mechanism transfers: they share far more surface form with each other than natural languages do, so identity is signalled by many correlated cues and often unresolved until several tokens in\. Whether a compact, steerable mechanism exists here is therefore an open question rather than a corollary, and to our knowledge this work and its earlier version are the first causal, cross\-model attempts to settle it\. Our corpora come from MultiPL\-E\([Cassano et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib4)\), XLCoST\([Zhu et al\., 2022](https://arxiv.org/html/2610.20683#bib.bib29)\), and CodeXGLUE\([Lu et al\., 2021](https://arxiv.org/html/2610.20683#bib.bib15)\)\. Sparse\-autoencoder decompositions\([Cunningham et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib6);[Bricken et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib3);[Templeton et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib22)\)are an alternative unit of analysis we return to in §[5](https://arxiv.org/html/2610.20683#S5)\.

## 3Measuring language preference

### 3\.1Prompts

Two prompt suites appear in this paper\. Both consist of*pairs*: a base member written in Python and a target member written in Rust, C\#, or PHP, which are identical in what the code does and differ in language\.

#### Aligned prompts\.

Our main suite is 64 structurally aligned function pairs built from four templates \(add,square,greet,is\_even\)\. Each prompt is a one\-shot context whose final line is a language tag, so the model’s next token is the keyword that opens a function body\. The base member is

> \# language: python def add\(a, b\): return a \+ b \# language: python →\\rightarrow\(next token scored here\)

and the target member is identical up to the tag and body, e\.g\.// language: rustfollowed byfn add\(a: i32, b: i32\) \-\> i32 \{ \.\.\. \}\. The design does one job: it forces a language decision at a known position, so the metric below is read where the model must commit\. This makes most other candidate tokens implausible there—after\# language: pythonthe plausible continuations are essentiallydefand a newline—which is what makes the measurement clean and, as Finding 2 shows, is also its main limitation\.

#### XLCoST prompts\.

To test whether the aligned suite manufactures its own answer, we rerun the identical pipeline on 128 parallel function pairs drawn from XLCoST\([Zhu et al\., 2022](https://arxiv.org/html/2610.20683#bib.bib29)\), a cross\-lingual code corpus of naturally occurring programs aligned across languages\. These prompts are real code, so the scored position is wherever the aligned pair happens to end rather than a constructed decision point\.

#### Held\-out behavioural prompts\.

Generation and downstream checks use a separate set of instruction prompts: two Python→\\rightarrowtarget translation tasks, a target\-language completion, and two controls \(an English→\\rightarrowFrench translation and a Python completion\) that a syntax\-specific intervention should leave alone\. Keeping these separate from the pairs stops the behavioural checks contaminating the causal estimates\.

### 3\.2A continuous preference metric

The language of a finished program is a discrete label, which gives an intervention almost nothing to move\. We instead read a continuous signal\. For a target language we fix a small setVtargetV\_\{\\mathrm\{target\}\}of tokens that open target\-language syntax, and a setVbaseV\_\{\\mathrm\{base\}\}that opens Python syntax \(Table[2](https://arxiv.org/html/2610.20683#S3.T2)\)\. For an inputxxwe score

ΔM​\(x\)=∑t∈Vtargetp⁡\(t∣x\)−∑t∈Vbasep⁡\(t∣x\)∈\[−1,1\]\\Delta\_\{M\}\(x\)=\\sum\_\{t\\in V\_\{\\mathrm\{target\}\}\}p\(t\\mid x\)\\;\-\\;\\sum\_\{t\\in V\_\{\\mathrm\{base\}\}\}p\(t\\mid x\)\\;\\in\\;\[\-1,1\]\(1\)
at the scored position\. Positive means the model is about to write the target language; negative means Python\. Surface strings are resolved to token ids by encoding each under several natural prefixes \(empty, leading space, newline, newline plus indent\) and keeping the first sub\-token whose decoded form matches, which recovers the id a model actually emits for, say,fnat the start of a line\. Ids appearing in both sets are dropped, soΔM\\Delta\_\{M\}never credits a language for syntax the two share\. For a pair\(xt,xb\)\(x\_\{t\},x\_\{b\}\)we summarise raw preference by the*separation*S=ΔM​\(xt\)−ΔM​\(xb\)S=\\Delta\_\{M\}\(x\_\{t\}\)\-\\Delta\_\{M\}\(x\_\{b\}\)\.

Table 2:Syntax vocabularies definingΔM\\Delta\_\{M\}\. Overlapping ids are removed at build time, so the two sets are disjoint\. C\# and PHP need more strings because no single keyword is as diagnostic for them asfnis for Rust; §[4\.2](https://arxiv.org/html/2610.20683#S4.SS2)shows this does not drive the results\.
### 3\.3Interventions, and the gate on every claim

We study five model families across a10×10\\timessize range—CodeLlama\-7B\([Rozière et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib19)\), DeepSeek\-Coder\-1\.3B/6\.7B\([Guo et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib10)\), and Qwen2\.5\-Coder\-7B/14B\([Hui et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib12)\)—in both base and instruction\-tuned form, against Rust, C\#, and PHP, through TransformerLens\-style hooks\([Nanda and Bloom, 2022](https://arxiv.org/html/2610.20683#bib.bib17)\)\. A componentccis an attention head or an MLP block, addressed uniformly as a \(kind, layer, head\) triple\. Letac​\(x\)a\_\{c\}\(x\)be its activation on inputxx\. Four operators act on Eq\.[1](https://arxiv.org/html/2610.20683#S3.E1)\.

#### Necessity\.

Mean\-ablate each component on the target prompt and rank by the resulting drop,δ\(c\)=ΔM\(xt\)−ΔM\(xt;ac←a¯c\)\\delta\(c\)=\\Delta\_\{M\}\(x\_\{t\}\)\-\\Delta\_\{M\}\(x\_\{t\};a\_\{c\}\\\!\\leftarrow\\\!\\bar\{a\}\_\{c\}\), wherea¯c\\bar\{a\}\_\{c\}is the component’s mean activation over the suite\. We carry the topkkcomponents forward\. Mean ablation, rather than zeroing, keeps the statistics the rest of the network expects\.kkis the budget that Finding 3 varies: we run the whole grid atk=12k=12and again atk=40k=40\.

#### Sufficiency\.

Transplant the target\-run activations of a candidate set𝒞\\mathcal\{C\}into the clean Python run and ask whether the sign changes:ΔMpatch=ΔM\(xb;ac←ac\(xt\)∀c∈𝒞\)\\Delta\_\{M\}^\{\\mathrm\{patch\}\}=\\Delta\_\{M\}\(x\_\{b\};\\,a\_\{c\}\\\!\\leftarrow\\\!a\_\{c\}\(x\_\{t\}\)\\;\\forall c\\in\\mathcal\{C\}\), with success requiringΔM​\(xb\)<0<ΔMpatch\\Delta\_\{M\}\(x\_\{b\}\)<0<\\Delta\_\{M\}^\{\\mathrm\{patch\}\}\.

#### Minimality\.

Rank candidates by their individual patched shift and add them greedily until the accumulated shift reachesτ=0\.90\\tau=0\.90of the full\-set effect\. This yields the extracted circuit and is a node\-level approximation of edge attribution patching\([Syed et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib20)\)\.

#### Specificity, and the gate on every claim\.

An extracted circuit is only interesting if it beats a circuit of the same size chosen at random\. We compare the circuit’s per\-pair patched shifts againstN=30N=30size\-matched random component sets with a Welchtt\-test\. Because we run this test in every setting, a raw threshold would over\-report; we control the false\-discovery rate with the Benjamini–Hochberg procedure within each probe grid and require the correctedqBH<0\.05q\_\{\\mathrm\{BH\}\}<0\.05\. We say a circuit is*specific*when it clears that threshold*and*its mean shift points toward the target language, and*away\-pointing*when it clears the threshold but points the other way\.

#### Steering\.

For each circuit component we form the mean target\-minus\-base direction and add it, scaled byα\\alpha, during the forward pass on a Python prompt\([Turner et al\., 2023](https://arxiv.org/html/2610.20683#bib.bib24)\):a~c=ac\+α​dc\\tilde\{a\}\_\{c\}=a\_\{c\}\+\\alpha\\,d\_\{c\}withdc=𝔼x​\[ac​\(xt\)−ac​\(xb\)\]d\_\{c\}=\\mathbb\{E\}\_\{x\}\[a\_\{c\}\(x\_\{t\}\)\-a\_\{c\}\(x\_\{b\}\)\]\. We sweepα∈\{0,1,2,4,8\}\\alpha\\in\\\{0,1,2,4,8\\\}and recordΔM\\Delta\_\{M\}, whether the most likely next token is target syntax, and downstream generation behaviour\.

## 4Results

### 4\.1The preference survives scale and instruction tuning

The sign ofΔM\\Delta\_\{M\}recovers the intended language in all 45 aligned\-prompt settings without exception: the target prompt always scores strongly positive and the base prompt strongly negative\. Separation grows with model size inside each family, from a meanSSof1\.511\.51at DeepSeek\-Coder\-1\.3B to1\.611\.61at 6\.7B, and from1\.731\.73at Qwen2\.5\-Coder\-7B to1\.811\.81at 14B\. Two sizes per family cannot establish a scaling law, but the direction is consistent\.

Instruction tuning does not remove the preference\. Figure[1](https://arxiv.org/html/2610.20683#S4.F1)pairs each base checkpoint with its instruction\-tuned sibling on the identical measurement\. The instruction\-tuned model has the*larger*separation in all 15 settings, by 12% on average and by as much as 30% \(DeepSeek\-Coder\-1\.3B on Rust,1\.49→1\.931\.49\\rightarrow 1\.93\)\. The base Python score also becomes more negative after tuning, meaning the tuned model is more confidently Python when the tag says Python\. This matters for motivation: whatever post\-training does to these models, it does not dissolve the phenomenon, so studying it on base models was not a scope error\.

BecauseSSis read at a language\-*tagged*position, a negative base score could reflect tag\-following instead of a real Python default\. Re\-measuringΔM\\Delta\_\{M\}on ten bare instruction prompts with no tag \(Appendix[B](https://arxiv.org/html/2610.20683#A2)\) leaves the default intact: it is Python\-leaning in 12 of 15 settings, with the three exceptions all PHP\. The effect is one to two orders of magnitude smaller thanSS, so the unconstrained prior is real but faint\.

Finding 1Language preference is large, grows with model size, and is*stronger*after instruction tuning, not weaker\.

Figure 1:SeparationSSfor each base checkpoint \(hollow bars\) and its instruction\-tuned sibling \(filled\), on the identical prompt suite\. Instruction tuning increasesSSin all 15 settings\. Per\-setting values are in Table[3](https://arxiv.org/html/2610.20683#S4.T3)\.
### 4\.2Vocabulary size does not drive the metric

Because Eq\.[1](https://arxiv.org/html/2610.20683#S3.E1)sums probabilities, a largerVtargetV\_\{\\mathrm\{target\}\}could inflateΔM\\Delta\_\{M\}mechanically, and the C\# and PHP sets are twice the size of the Rust set after id resolution \(12 ids vs\. 6; the Python set is 6 in all cases\)\. Three observations bound this concern\. First, the base term is identical across the three target languages within a model, so only the target term can vary\. Second, doubling the vocabulary buys very little: averaged over the five base models, meanSSis1\.6031\.603for Rust,1\.6501\.650for C\#, and1\.6631\.663for PHP—a gap of0\.060\.06, or 3\.7% ofSS, against a2×2\\timesdifference in set size\. Third, across the 15 aligned base settings, set size and separation are not associated \(Spearmanρ=\+0\.20\\rho=\+0\.20,p=0\.48p=0\.48\)\. The scores are also near saturation \(ΔM​\(xt\)∈\[0\.76,0\.90\]\\Delta\_\{M\}\(x\_\{t\}\)\\in\[0\.76,0\.90\]\), which caps how much extra mass more tokens can collect\. A fully size\-matched vocabulary would still be the cleaner design, and we flag it as unfinished work in §[6](https://arxiv.org/html/2610.20683#S6)\.

### 4\.3The measured size belongs to the probe, not to the model

Rerunning the identical pipeline on XLCoST parallel functions gives a different picture of the same models\. Separation collapses from≈1\.6\\approx 1\.6to below0\.0120\.012in every one of the ten settings—two orders of magnitude \(Table[3](https://arxiv.org/html/2610.20683#S4.T3)\)\. Both terms of Eq\.[1](https://arxiv.org/html/2610.20683#S3.E1)are near zero: on naturally occurring code, at the position where an aligned pair happens to end, the model is not about to emit a language\-defining keyword at all, so almost no probability mass sits on either vocabulary\.

We read this as a statement about the probe, not about the models\. The aligned suite does not measure defaultness in the wild; it constructs a position where a language decision must be made and reads the model’s commitment there\. That is what makes the causal interventions interpretable, but the number it produces is not the size of the bias a developer meets\. Consistent with this, no XLCoST setting yields a specific circuit and no intervention moves anything: with the metric already at zero there is nothing to move\.

Table 3:Language preference under all four probes\.ΔM​\(xt\)\\Delta\_\{M\}\(x\_\{t\}\)scores the target\-language prompt andSSis the separation between the target and base prompts \(§[3\.2](https://arxiv.org/html/2610.20683#S3.SS2)\)\. “No tag” is the meanΔM\\Delta\_\{M\}over ten bare instruction prompts carrying no language tag, where negative means Python\-leaning\. XLCoST was run for C\# and PHP only\.Finding 2The measured size of the preference belongs to the probe: the same metric on the same models reads≈1\.6\\approx 1\.6on aligned prompts and≈0\.01\\approx 0\.01on XLCoST\.

### 4\.4Localisation depends on the component budget

Keeping the top 12 components after ablation ranking was never justified by anything except tractability, yet it carries the earlier version’s central negative claim\. Table[4](https://arxiv.org/html/2610.20683#S4.T4)gives the specificity test at both budgets\. Atk=12k=12, exactly one setting \(CodeLlama\-7B on Rust\) beats size\-matched random controls after correction\. Atk=40k=40—the range used by sparse\-feature circuit work\([Marks et al\., 2025](https://arxiv.org/html/2610.20683#bib.bib31)\)—six do, and two of them are large: DeepSeek\-Coder\-1\.3B/C\# shiftsΔM\\Delta\_\{M\}by\+0\.61\+0\.61and Qwen2\.5\-Coder\-7B/PHP by\+1\.25\+1\.25, against random\-control means below0\.030\.03\. On the instruction\-tuned grid atk=12k=12, two settings are specific, one of them \(\+1\.27\+1\.27\) larger than anything on the base models\.

Only CodeLlama\-7B/Rust is specific at both budgets, so the identity of the surviving settings is not stable either\. The circuits share no structure across models: they span the full depth range \(L9MLPin CodeLlama\-7B,L12MLPin Qwen2\.5\-Coder\-14B,L22\-\-L24MLPin Qwen2\.5\-Coder\-7B,L29MLPin DeepSeek\-Coder\-6\.7B\), and they are MLP\-mediated: attention heads appear in the ablation rankings but rarely survive extraction\. Circuit size moves with the budget too—atk=40k=40, four settings return the full candidate set and four others return a single component \(Appendix[D](https://arxiv.org/html/2610.20683#A4)\)\.

The honest summary is not “the preference is localisable” or “it is not”\. It is that a study reporting either answer is reporting a property of its search budget as much as of the model, and that budgets in this literature are chosen by convention\.

Table 4:Specificity across component budgets\. “shift” is the circuit’s mean patched shift inΔM\\Delta\_\{M\};qBHq\_\{\\mathrm\{BH\}\}corrects the Welch test against 30 size\-matched random sets within each column pair\.∗specific;†significant but pointing away from the target language\.Finding 3Whether the preference looks localised depends on the search budget: going from 12 components to 40 takes the count of specific circuits from 1 of 15 to 6 of 15\.

### 4\.5Patching and steering work in a minority of settings

#### Patching\.

Transplanting the extracted components from the target run into a Python prompt movesΔM\\Delta\_\{M\}across zero in 2 of the 45 aligned and instruction\-tuned settings \(Table[5](https://arxiv.org/html/2610.20683#S4.T5), left\)\. Both are Qwen2\.5\-Coder\-7B: the base model on PHP atk=40k=40\(−0\.91→\+0\.50\-0\.91\\rightarrow\+0\.50, circuitL22\-\-L24MLPplusL27H13\) and the instruction\-tuned model on Rust atk=12k=12\(−0\.99→\+0\.50\-0\.99\\rightarrow\+0\.50, a 12\-component set led byL24MLP\)\. Both circuits are also specific by the test of §[4\.4](https://arxiv.org/html/2610.20683#S4.SS4), so the two results agree\. Elsewhere the patched metric stays negative; the next closest is DeepSeek\-Coder\-1\.3B/C\# at−0\.14\-0\.14and every other setting stays below−0\.6\-0\.6, so these are not near misses\.

#### Steering\.

Adding the difference\-in\-means direction changes the most likely next token to target syntax in 9 of 45 settings: 6 atk=40k=40on base models and 3 on instruction\-tuned models \(Table[5](https://arxiv.org/html/2610.20683#S4.T5), right; Figure[2](https://arxiv.org/html/2610.20683#S4.F2)\)\. Atk=12k=12on base models it never does, which is what the earlier version reported\. The coefficient matters\. Three of the nine switch atα=1\\alpha=1, two of them the settings where patching also worked; the remaining six needα≥2\\alpha\\geq 2and four needα≥4\\alpha\\geq 4\. That upper range is where behaviour degrades: generation becomes repetitive, and on Qwen2\.5\-Coder\-14B/C\# a coefficient of 4 or more injects 126–128 C\# markers into control prompts that should contain none\. A safe coefficient exists but must be searched for per setting: for Qwen2\.5\-Coder\-7B it is 8 on Rust, 4 on PHP, and 0 on C\#, so C\# cannot be steered safely at any strength we test \(Appendices[G](https://arxiv.org/html/2610.20683#A7),[H](https://arxiv.org/html/2610.20683#A8)\)\.

#### Reading a metric flip correctly\.

A curve crossing zero is tempting to read as a switch, butΔM\\Delta\_\{M\}is a difference of probability masses: it rises either by moving mass onto target keywords or by flattening the distribution\. Two checks separate these\. The argmax check—is the most likely token target syntax?—is the one reported above\. The contamination check asks whether the same coefficient injects target markers into unrelated prompts, and it fails for Qwen2\.5\-Coder\-14B/C\# atα≥4\\alpha\\geq 4\. A defensible claim needs a metric shift, a target argmax, and clean controls together\. Nine settings give the first two; the coefficient window in which they give all three is narrow and model\-specific\.

Table 5:Interventions on the Python prompt\. Left:ΔM\\Delta\_\{M\}after patching the extracted circuit; the clean value before patching isΔM​\(xt\)−S\\Delta\_\{M\}\(x\_\{t\}\)\-Sfrom Table[3](https://arxiv.org/html/2610.20683#S4.T3), and bold marks the two runs that cross zero\. Right: the smallest steering coefficient at which the most likely next token becomes target syntax; “—” means none of\{1,2,4,8\}\\\{1,2,4,8\\\}does\.ModelTargetPatchedΔM\\Delta\_\{M\}\(clean→\\rightarrowpatched\)Smallestα\\alphawith target argmaxkk=12kk=40instr\.kk=12kk=40instr\.CodeLlama\-7BRust\-0\.686\-0\.692\-0\.761———C\#\-0\.979\-0\.987\-0\.991———PHP\-0\.738\-0\.668\-0\.751——2DeepSeek\-Coder\-1\.3BRust\-0\.718\-0\.742\-0\.924—44C\#\-0\.752\-0\.136\-0\.939—1—PHP\-0\.695\-0\.637\-0\.970———DeepSeek\-Coder\-6\.7BRust\-0\.732\-0\.811\-0\.790———C\#\-0\.773\-0\.745\-0\.797—8—PHP\-0\.793\-0\.739\-0\.854—4—Qwen2\.5\-Coder\-7BRust\-0\.914\-0\.898\+0\.504——1C\#\-0\.859\-0\.863\-0\.992———PHP\-0\.906\+0\.500\-0\.992—1—Qwen2\.5\-Coder\-14BRust\-0\.949\-0\.945\-0\.980———C\#\-0\.949\-0\.914\-0\.973———PHP\-0\.891\-0\.840\-0\.965—2—Figure 2:Steering sweeps atk=40k=40on the base models, one panel per target language\. Filled markers are coefficients at which the most likely next token has become target syntax\. The metric moves well before the argmax does, and in most settings the argmax never follows\.Finding 4Patching and steering are not uniformly inert—they work in 2 of 45 and 9 of 45 settings—but mostly at coefficients large enough to damage the output\.

### 4\.6No single component is behaviourally load\-bearing

Starting from the CodeLlama\-7B/Rust set atα=1\\alpha=1, we remove one component at a time and re\-score the explicit Rust and control instruction suite\. Every removal—including dropping the specific circuit itself—leaves marker accuracy and F1 at1\.0001\.000, and mean next\-tokenΔM\\Delta\_\{M\}moves by less than0\.020\.02\(Appendix[I](https://arxiv.org/html/2610.20683#A9)\)\. Explicit prompts carry so many correlated language cues that no single steering component changes what the model does\. Two CodeXGLUE tasks\([Lu et al\., 2021](https://arxiv.org/html/2610.20683#bib.bib15)\)stay pinned at chance under steering, so the direction encodes surface syntax, not task competence \(Appendix[J](https://arxiv.org/html/2610.20683#A10)\)\.

### 4\.7Circuits that point away are extraction failures

Five results across three settings clear the significance threshold with the shift pointing the wrong way; the largest, CodeLlama\-7B on C\# at−0\.43\-0\.43, reproduces at both budgets and on the instruction\-tuned checkpoint\. Transplanting C\# activations into a Python prompt makes the model*more*confidently Python\. Two observations explain it, and neither is about C\#\.

First, transplanting a late\-layer activation is damaging in every language\. Figure[3](https://arxiv.org/html/2610.20683#A3.F3)sums each component’s individual patched shift by depth band in CodeLlama\-7B\. Early components contribute close to nothing \(\+0\.06\+0\.06,\+0\.01\+0\.01,\+0\.08\+0\.08for Rust, C\# and PHP\); middle and late components contribute large negative shifts, the late band with only four or five components\. A late\-layer activation is computed for a specific context—part\-way through a target\-language function body—so transplanting it into a prompt sitting just after a Python language tag inserts a state the network never produces, and the model falls back to its most confident generic continuation, which after\# language: pythonis Python syntax\. Summed over all 40 components the damage is nearly identical across the three languages \(−0\.46\-0\.46,−0\.43\-0\.43,−0\.46\-0\.46\), so it cannot be what separates them\.

Second, what separates them is whether extraction keeps those components\. For CodeLlama\-7B, greedy extraction returns the single componentL9MLPfor Rust and for PHP, but for C\# it finds no compact subset and returns all 40 candidates, including every damaging late MLP \(Table[7](https://arxiv.org/html/2610.20683#A4.T7)\)\. The pattern holds across the grid: all five away\-pointing results come from settings where extraction returned the full candidate set, and atk=40k=40full\-set circuits have a median shift of−0\.04\-0\.04against\+0\.05\+0\.05for compact ones\. A significant negative result is therefore a warning about the search, not evidence of an anti\-circuit, and such searches should report extracted size alongside significance\.

Finding 5Every significant*away*\-pointing circuit comes from a setting where extraction failed to find a compact subset\. It diagnoses the search, not the model\.

## 5Discussion

#### For practitioners\.

The preference is real and cheap to monitor—one forward pass—but it is not a reliable dial\. Steering changes the next token in a fifth of settings, needs a coefficient searched per setting, and trades syntax for fluency past that point\. Prompting and fine\-tuning remain the dependable route\.

#### For interpretability research\.

This is the result we think travels\. Two knobs normally reported as setup—how many components the search keeps, and which prompts the metric is read on—moved our headline conclusions from “not localisable, never steerable” to “localisable in 6 of 15 settings, steerable in 9 of 45”, and from “a large, robust bias” to “a bias two orders of magnitude smaller”\. Neither knob is exotic\. Circuit papers should report a budget sweep the way ablation papers report a seed sweep, and state a localisation claim as a curve over budgets rather than a single number\. Where a handful of neurons can toggle a multilingual model’s output language\([Tang et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib21);[Kojima et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib13)\), we find no comparably compact structure for code: identity there is signalled by many correlated cues, so no component is a bottleneck\. Sparse\-feature\([Cunningham et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib6);[Marks et al\., 2025](https://arxiv.org/html/2610.20683#bib.bib31)\)or edge\-level\([Syed et al\., 2024](https://arxiv.org/html/2610.20683#bib.bib20)\)analyses are the natural next step\.

## 6Limitations

Metric\.ΔM\\Delta\_\{M\}uses a small hand\-chosen vocabulary read at one position; §[4\.2](https://arxiv.org/html/2610.20683#S4.SS2)bounds the effect of set size without removing it, and the C\# and PHP sets share markers with other C\-family languages\.Method\.Minimality is node\-level and approximates edge attribution patching, and steering adds one difference\-in\-means direction, so Finding 4 bounds these methods rather than all possible controls\.Scope\.We test two budgets, one base language, three targets, and two sizes per family—enough to show that budget matters, not enough for a scaling law\.

## 7Conclusion

Code models’ preference for Python is large, grows with scale, and survives instruction tuning, but the mechanistic story about it is not stable under choices that look like setup\. Raising the budget from 12 components to 40 takes the number of settings with a statistically specific circuit from 1 to 6; swapping the prompt suite takes the measured preference from1\.61\.6to0\.010\.01; and every away\-pointing circuit marks a failed search\. We release the grid and the pipeline so these sweeps can be continued rather than assumed\.

## References

- Arditiet al\.\(2024\)A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. NandaRefusal in language models is mediated by a single direction\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.20683#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.
- Bereska and Gavves \(2024\)L\. Bereska and E\. GavvesMechanistic interpretability for AI safety – a review\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Brickenet al\.\(2023\)T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly,et al\.Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1)\.
- Cassanoet al\.\(2023\)F\. Cassano, J\. Gouwar, D\. Nguyen, S\. Nguyen, L\. Phipps\-Costin, D\. Pinckney,et al\.MultiPL\-E: a scalable and polyglot approach to benchmarking neural code generation\.IEEE Transactions on Software Engineering49\(7\),pp\. 3675–3691\.Cited by:[§1](https://arxiv.org/html/2610.20683#S1.p1.1),[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1)\.
- Conmyet al\.\(2023\)A\. Conmy, A\. N\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-AlonsoTowards automated circuit discovery for mechanistic interpretability\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Cunninghamet al\.\(2024\)H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. SharkeySparse autoencoders find highly interpretable features in language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.20683#S5.SS0.SSS0.Px2.p1.1)\.
- Elhageet al\.\(2022\)N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec,et al\.Toy models of superposition\.Transformer Circuits Thread\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.
- Geigeret al\.\(2021\)A\. Geiger, H\. Lu, T\. Icard, and C\. PottsCausal abstractions of neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Geigeret al\.\(2024\)A\. Geiger, Z\. Wu, C\. Potts, T\. Icard, and N\. D\. GoodmanFinding alignments between interpretable causal variables and distributed neural representations\.InCausal Learning and Reasoning \(CLeaR\),pp\. 160–187\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2024\)D\. Guo, Q\. Zhu, D\. Yang, Z\. Xie, K\. Dong, W\. Zhang,et al\.DeepSeek\-Coder: when the large language model meets programming – the rise of code intelligence\.arXiv preprint arXiv:2401\.14196\.Cited by:[§3\.3](https://arxiv.org/html/2610.20683#S3.SS3.p1.1)\.
- Hendelet al\.\(2023\)R\. Hendel, M\. Geva, and A\. GlobersonIn\-context learning creates task vectors\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 9318–9333\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.
- Huiet al\.\(2024\)B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang,et al\.Qwen2\.5\-Coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§3\.3](https://arxiv.org/html/2610.20683#S3.SS3.p1.1)\.
- Kojimaet al\.\(2024\)T\. Kojima, I\. Okimura, Y\. Iwasawa, H\. Yanaka, and Y\. MatsuoOn the multilingual ability of decoder\-based pre\-trained language models: finding and controlling language\-specific neurons\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics,Cited by:[§1](https://arxiv.org/html/2610.20683#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.20683#S5.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2023\)K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. WattenbergInference\-time intervention: eliciting truthful answers from a language model\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.
- Luet al\.\(2021\)S\. Lu, D\. Guo, S\. Ren, J\. Huang, A\. Svyatkovskiy, A\. Blanco,et al\.CodeXGLUE: a machine learning benchmark dataset for code understanding and generation\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Cited by:[Table 13](https://arxiv.org/html/2610.20683#A10.T13),[Appendix J](https://arxiv.org/html/2610.20683#A10.p1.1),[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1),[§4\.6](https://arxiv.org/html/2610.20683#S4.SS6.p1.1)\.
- Markset al\.\(2025\)S\. Marks, C\. Rager, E\. J\. Michaud, Y\. Belinkov, D\. Bau, and A\. MuellerSparse feature circuits: discovering and editing interpretable causal graphs in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.20683#S1.SS0.SSS0.Px3.p1.1),[§4\.4](https://arxiv.org/html/2610.20683#S4.SS4.p1.1),[§5](https://arxiv.org/html/2610.20683#S5.SS0.SSS0.Px2.p1.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Nanda and Bloom \(2022\)N\. Nanda and J\. BloomTransformerLens\.Note:[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)Cited by:[§3\.3](https://arxiv.org/html/2610.20683#S3.SS3.p1.1)\.
- Parket al\.\(2024\)K\. Park, Y\. J\. Choe, and V\. VeitchThe linear representation hypothesis and the geometry of large language models\.InInternational Conference on Machine Learning,pp\. 39643–39666\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.
- Rozièreet al\.\(2023\)B\. Rozière, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan,et al\.Code Llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§3\.3](https://arxiv.org/html/2610.20683#S3.SS3.p1.1)\.
- Syedet al\.\(2024\)A\. Syed, C\. Rager, and A\. ConmyAttribution patching outperforms automated circuit discovery\.InProceedings of the 7th BlackboxNLP Workshop,pp\. 407–416\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2610.20683#S3.SS3.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.20683#S5.SS0.SSS0.Px2.p1.1)\.
- Tanget al\.\(2024\)T\. Tang, W\. Luo, H\. Huang, D\. Zhang, X\. Wang, X\. Zhao, F\. Wei, and J\. WenLanguage\-specific neurons: the key to multilingual capabilities in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 5701–5715\.Cited by:[§1](https://arxiv.org/html/2610.20683#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.20683#S5.SS0.SSS0.Px2.p1.1)\.
- Templetonet al\.\(2024\)A\. Templeton, T\. Conerly, J\. Marcus, J\. Batson, B\. Rosen, H\. Cunningham,et al\.Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet\.Transformer Circuits Thread\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1)\.
- Toddet al\.\(2024\)E\. Todd, M\. L\. Li, A\. S\. Sharma, A\. Mueller, B\. C\. Wallace, and D\. BauFunction vectors in large language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.
- Turneret al\.\(2023\)A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmidSteering language models with activation engineering\.arXiv preprint arXiv:2308\.10248\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2610.20683#S3.SS3.SSS0.Px5.p1.1)\.
- Viget al\.\(2020\)J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, Y\. Singer, and S\. ShieberCausal mediation analysis for interpreting neural NLP: the case of gender bias\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)K\. R\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. SteinhardtInterpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Zhang and Nanda \(2024\)F\. Zhang and N\. NandaTowards best practices of activation patching in language models: metrics and methods\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2023\)Q\. Zheng, X\. Xia, X\. Zou, Y\. Dong, S\. Wang, Y\. Xue,et al\.CodeGeeX: a pre\-trained model for code generation with multilingual benchmarking on HumanEval\-X\.arXiv preprint arXiv:2303\.17568\.Cited by:[§1](https://arxiv.org/html/2610.20683#S1.p1.1)\.
- Zhuet al\.\(2022\)M\. Zhu, A\. Jain, K\. Suresh, R\. Ravindran, S\. Tipirneni, and C\. K\. ReddyXLCoST: a benchmark dataset for cross\-lingual code intelligence\.arXiv preprint arXiv:2206\.08474\.Cited by:[§1](https://arxiv.org/html/2610.20683#S1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2610.20683#S3.SS1.SSS0.Px2.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren,et al\.Representation engineering: a top\-down approach to AI transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§2](https://arxiv.org/html/2610.20683#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix

## Appendix AConfiguration and implementation

All runs share the settings in Table[6](https://arxiv.org/html/2610.20683#A1.T6)\. CodeLlama uses a TransformerLensHookedTransformerbackend; DeepSeek\-Coder and Qwen2\.5\-Coder, which are not in that registry, use the TransformerLens bridge backend, which wraps a Hugging Face model and re\-exposes the sameblocks\.\{layer\}\.attn\.hook\_zandblocks\.\{layer\}\.hook\_mlp\_outhook points, so one implementation of every intervention runs unchanged across backends\. Attention\-head interventions act on the per\-headhook\_ztensor and MLP interventions onhook\_mlp\_out\. The mean\-ablation cache is built once per run by averaging each hook’s activation over the prompt suite with the position axis reduced, so a single stored vector broadcasts over any sequence length\. Qwen2\.5\-Coder produced NaN logits in fp16 on the bridge path, so all Qwen runs use bf16 and the others use fp16\. Each run, together with its generation and CodeXGLUE evaluations, completes within a few GPU\-hours on one A100\-80GB\.

Table 6:Shared experimental configuration\.
## Appendix BUntagged control

The “No tag” column of Table[3](https://arxiv.org/html/2610.20683#S4.T3)reports the meanΔM\\Delta\_\{M\}over ten bare instruction prompts \(for example, “Write a function that adds two numbers\.”\) with no\# languagetag\. Negative means Python\-leaning\. Twelve of fifteen settings stay negative and the three exceptions are all PHP, which is consistent with PHP’s markers overcoming a weak untagged prior in the smaller models\. The magnitudes are one to two orders below the tagged separation, so the unconstrained default is real but faint\. The Qwen family shows a markedly stronger untagged default than CodeLlama or DeepSeek, and within Qwen the 7B is stronger than the 14B, so unlikeSSthe untagged effect does not grow monotonically with scale\.

## Appendix CComponent depth and away\-pointing shifts

Figure[3](https://arxiv.org/html/2610.20683#A3.F3)is the evidence behind §[4\.7](https://arxiv.org/html/2610.20683#S4.SS7)\. It sums the individual patched shift of every candidate component by depth band in CodeLlama\-7B, the model with the largest away\-pointing result\. Summing rather than averaging is the right summary here because the combined patch transplants all of them at once; note, though, that patching is not additive, so these sums do not equal the combined shifts in Table[5](https://arxiv.org/html/2610.20683#S4.T5)\.

Figure 3:Summed individual patched shift of thek=40k=40candidate components in CodeLlama\-7B, by depth band; numbers above or below each bar give how many components fall in that band\. Early components are harmless, and the four to five late components carry as much negative shift as the whole middle band\. The totals are nearly identical across the three languages, so depth explains the damage but not which setting ends up away\-pointing \(§[4\.7](https://arxiv.org/html/2610.20683#S4.SS7)\)\.
## Appendix DExtracted circuits per setting

Table[7](https://arxiv.org/html/2610.20683#A4.T7)gives the circuit that greedy extraction returns in each setting\. A small circuit does not by itself imply specificity: several single\-component circuits in Table[7](https://arxiv.org/html/2610.20683#A4.T7)fail the test in Table[4](https://arxiv.org/html/2610.20683#S4.T4)\. The layers chosen share no pattern across models, which is further evidence against a transferable syntax circuit\.

Table 7:Circuit returned by greedy extraction \(§[3](https://arxiv.org/html/2610.20683#S3)\) per setting\. “Full candidate set” means no compact subset reached 90% of the full\-set effect\.
## Appendix EFull steering sweep

Table[8](https://arxiv.org/html/2610.20683#A5.T8)gives theΔM\\Delta\_\{M\}value at every coefficient for thek=40k=40base\-model grid plotted in Figure[2](https://arxiv.org/html/2610.20683#S4.F2)\. Bold entries are coefficients at which the most likely next token is target syntax\. Reading the table alongside Appendices[G](https://arxiv.org/html/2610.20683#A7)and[H](https://arxiv.org/html/2610.20683#A8)shows the problem with treating a metric crossing as a switch: several crossings occur at coefficients where generation has already degraded\.

Table 8:ΔM\\Delta\_\{M\}on the Python prompt across the steering sweep, base models atk=40k=40\. Bold marks coefficients where the most likely next token is target syntax\.
## Appendix FXLCoST detail

Table[9](https://arxiv.org/html/2610.20683#A6.T9)separates the two terms ofΔM\\Delta\_\{M\}on the XLCoST suite\. Both are near zero, which is why the separation collapses: at the scored position in naturally occurring code the model is not choosing between a Python and a target\-language keyword, so almost no probability mass falls on either vocabulary\. The comparison column repeats the aligned\-prompt separation for the same model and language\.

Table 9:XLCoST prompts: both terms ofΔM\\Delta\_\{M\}, the resulting separation, and the aligned\-prompt separation for the same setting\.
## Appendix GSafe steering window

We call a coefficient*safe*when it classifies every target prompt as target\-language and no control prompt as target\-language\. Table[10](https://arxiv.org/html/2610.20683#A7.T10)reports the largest safe coefficient per setting for the two Qwen models\. The window is strongly model\- and language\-dependent, which is the practical counterpart of the negative steering result: no single strength is safe everywhere\. Qwen2\.5\-Coder\-7B toleratesα=8\\alpha=8on Rust andα=4\\alpha=4on PHP, while its C\# window collapses toα=0\\alpha=0\. At 14B the pattern shifts again: Rust is best left unsteered, while C\# and PHP have a genuine but narrow window atα=2\\alpha=2\.

Table 10:Largest safe steering coefficient per setting\. “Control FP” is the rate at which control prompts are wrongly classified as target\-language\.
## Appendix HControl contamination

Table[11](https://arxiv.org/html/2610.20683#A8.T11)sweepsα\\alphaon Qwen2\.5\-Coder\-14B and reports the target\-prompt success rate together with the rate at which control prompts are classified as target\-language\. C\# exposes the failure most sharply: atα=2\\alpha=2steering is effective and clean, but atα≥4\\alpha\\geq 4it injects 126–128 C\# markers into prompts that should contain none\. The apparent metric win at highα\\alphais an indiscriminate push, not language control\.

Table 11:Qwen2\.5\-Coder\-14B control\-contamination stress test\.Targetα\\alphaTarget successControl FPTarget markers in controlsRust01\.000\.000Rust40\.000\.000C\#21\.000\.000C\#41\.001\.00126C\#81\.001\.00128PHP21\.000\.002PHP81\.000\.506
## Appendix ILeave\-one\-out stress test

Starting from the CodeLlama\-7B/Rust top\-12 set atα=1\\alpha=1, we remove one component at a time and re\-score the explicit Rust and control instruction suite\. Every removal, including dropping the one specific circuit, leaves marker accuracy and F1 at1\.0001\.000, and the mean next\-tokenΔM\\Delta\_\{M\}stays within±0\.02\\pm 0\.02of the full\-set value \(Table[12](https://arxiv.org/html/2610.20683#A9.T12)\)\. No single component is behaviourally load\-bearing, which is what a redundant, multi\-cue encoding predicts\.

Table 12:CodeLlama\-7B/Rust leave\-one\-out steering atα=1\\alpha=1\(representative rows; all twelve removals give accuracy/F1=1\.000=1\.000\)\.
## Appendix JCodeXGLUE under steering

We cast two CodeXGLUE tasks\[[Lu et al\., 2021](https://arxiv.org/html/2610.20683#bib.bib15)\], big\-clone\-bench clone detection and defect detection, as zero\-shot yes/no next\-token decisions: the prompt states the task and ends in “Answer:”, and we compare the summed probability of affirmative label tokens against negative ones on class\-balanced samples, so0\.500\.50is chance\. This is a negative control\. If the steering direction encoded task competence rather than surface syntax, steering would move these scores\. It does not \(Table[13](https://arxiv.org/html/2610.20683#A10.T13)\)\. The only movement is atα=8\\alpha=8on defect detection, where the label bias shifts and macro\-F1 rises to0\.4910\.491— a redistribution of a degenerate predictor, not an accuracy gain\.

Table 13:Zero\-shot CodeXGLUE\[[Lu et al\., 2021](https://arxiv.org/html/2610.20683#bib.bib15)\]under steering\. Accuracy stays at chance; theα=8\\alpha=8defect shift is a label\-bias artifact\.