Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languages
Summary
This paper investigates how language models execute in-context conditional rules by probing whether testing and routing are separable mechanisms. Using activation patching across three open models and six languages, the authors find that predicate testing is modular while route representations are token-bound and non-transferable.
View Cached Full Text
Cached at: 08/06/26, 07:45 AM
# How Language Models Execute In-Context Conditional Rules Across Models and Languages
Source: [https://arxiv.org/html/2608.04183](https://arxiv.org/html/2608.04183)
Luxshan Thavarasa1,Sivasuthan Sukumar2 1Independent Researcher, Colombo, Sri Lanka 2Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka luxshanlux2000@gmail\.com,sivasuthan\.26@cse\.mrt\.ac\.lk
###### Abstract
When a language model follows an in\-context conditional rule such as “ifP\(x\)P\(x\)thenAAelseBB,” does it assemble a runtime circuit with one module that*tests*the predicate and another that*routes*the answer? We probe this with activation patching under a four\-donor design whose two swapped\-rule donors make the condition and the answer word*disagree*, so each layer reveals which of the two it carries\. Across three open models from two families and six languages sharing one fixed item bank, a mid\-stack residual band carries the predicate’s truth value: patching it reroutes the answer with predicate\-outcome flip near1\.01\.0and mapping flip near0\.00\.0, meeting a strict pre\-specified isolation criterion in 17 of 18 cells, and the same localization holds across five predicate families\. The router shows the opposite profile\. A learned subspace flipsA↔BA\{\\leftrightarrow\}Bnear\-perfectly within the trained pair yet transfers to a new pair at≈0\{\\approx\}0in every model, while in Gemma\-3\-4B \(the only model probed cross\-lingually\) it transfers at≈0\.98\{\\approx\}0\.98to the*same*pair in other languages\. Under every probe we ran, the router direction is token\-bound and non\-transferable \(largely answer\-readout in Gemma, pair\-specific in Qwen\) rather than an abstract routing module\. Test is modular; under these probes, route is not\.
Test, then Route: How Language Models Execute In\-Context Conditional Rules Across Models and Languages
Luxshan Thavarasa1, Sivasuthan Sukumar21Independent Researcher, Colombo, Sri Lanka2Department of Computer Science and Engineering, University of Moratuwa, Sri Lankaluxshanlux2000@gmail\.com,sivasuthan\.26@cse\.mrt\.ac\.lk
## 1Introduction
A large language model given an in\-context conditional rule \(“ifP\(x\)P\(x\)thenAAelseBB”\) must do two things to answer a query:*test*whether the predicateP\(x\)P\(x\)holds, and*route*the appropriate answer,AAorBB, to the output\. Whether the model assembles these as separable parts, or entangles testing and routing in one indivisible mechanism, is a basic question about how in\-context rules are executed, and one that behavioral evaluation cannot answer: a correct completion is consistent with either organization\.
Most mechanistic circuits characterized to date are*weight\-stored*: pretrained, reusable mechanisms invoked for a fixed task, such as the Indirect Object Identification circuit\(Wang et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib34)\)or induction heads\(Olsson et al\.,[2022](https://arxiv.org/html/2608.04183#bib.bib23)\)\. The conditional rule is different: its text, predicate, and answer mapping are all supplied at inference time and vary from prompt to prompt, so any circuit executing it must be*assembled at runtime*\. This places our object of study alongside function and task vectors\(Todd et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib31); Hendel et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib13)\)and filter heads\(Sharma et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib26)\), but shifts the question from*what*in\-context content is represented to*how*a two\-part conditional computation is factored across the residual stream\. Deployed prompts are dense with such conditionals \(guardrails, tool\-routing, conditional policies; Appendix[B](https://arxiv.org/html/2608.04183#A2)\), and which half is modular determines what can be audited and steered portably\.
We study a canonical rule \(“if the number is greater than 5, output dog, otherwise cat”\), presented with eight demonstrations and a numeric query whose single\-token answer the model completes\. Our central move is a set of four donor prompts \(Figure[2](https://arxiv.org/html/2608.04183#S3.F2)\) that, through activation patching, read off*per layer*whether a residual\-stream site carries the predicate’s truth*condition*or the routed*answer*word\. The two critical donors make the condition and the answer*disagree*: themap\-donorswaps the rule’s answer mapping while holding the predicate true; themap\-nulldoes so while holding it false\. A layer that follows the truth value holds the condition; one that follows the answer word holds the readout\. This exposes a sharp stage boundary \(Figure[3](https://arxiv.org/html/2608.04183#S4.F3)\): a mid\-stackteststage where patches follow the condition, and a laterroutestage where they follow the answer\.
Applying this, we find a clear asymmetry\. The predicate\-testis a causally separable, localizable module: patching a single mid\-stack band flips the answer by flipping only the predicate’s truth value\. Under every probe we ran, the argument\-routeshows no separable, transferable subspace: the direction that appears to route the answer flips one label pair but does not transfer to another\.*Test is modular; route, under these probes, is not\.*
Our contributions are thecondition\-vs\-answer disagreement designitself \(Figures[2](https://arxiv.org/html/2608.04183#S3.F2),[3](https://arxiv.org/html/2608.04183#S4.F3)\), run as a shared\-bank multilingual protocol: one fixed item bank rendered fully in each of six languages, with only the numerals and answer tokens held fixed\. Applying it yields aseparable predicate\-testband, localized over two label pairs, a three\-output variant, and multi\-token answers, and reproducing across three models from two families, six languages, and five predicates; and anon\-separableroute: a learned subspace \(distributed alignment search, DAS\) with a transfer control flips the trained label pair in every model yet does not transfer to a new pair \(in Gemma\-3\-4B it does transfer across languages\), a token\-bound readout direction rather than a reusable router; without the transfer control, the within\-pair interchange accuracy of1\.001\.00would have certified a spurious router in all three models\. An exploratory contrast with the weight\-stored IOI circuit appears in Appendix[C](https://arxiv.org/html/2608.04183#A3)\. Code, data, and per\-cell results will be released\.111[https://github\.com/Luxshan2000/icl\-conditional\-circuits](https://github.com/Luxshan2000/icl-conditional-circuits)
## 2Related Work
#### Circuits and activation patching\.
Mechanistic\-interpretability work localizes behavior to sparse subgraphs of attention heads and MLPs by causal intervention: indirect\-object identification set the template for reading off a task\-specific circuit\(Wang et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib34)\), activation\- and path\-based patching supply the causal\-mediation machinery\(Meng et al\.,[2022](https://arxiv.org/html/2608.04183#bib.bib20); Goldowsky\-Dill et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib10); Vig et al\.,[2020](https://arxiv.org/html/2608.04183#bib.bib33)\), and circuits can be reused across tasks\(Merullo et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib21)\)\. These circuits are predominantly*weight\-stored*: pretrained structures that a prompt merely activates\. Stage structure in in\-context tasks is also emerging: a universal mid\-stack request→\\toexecution split\(Variengien and Winsor,[2025](https://arxiv.org/html/2608.04183#bib.bib32)\)and modular sub\-circuits for in\-context propositional inference\(Hong et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib14); Kim et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib15); Chen et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib3)\)\. We decompose an in\-context*conditional*\(predicate evaluation vs\. answer routing with an explicit else branch\) under donors that make condition and answer disagree\.
#### In\-context learning as vectors, subspaces, and functions\.
A complementary tradition compresses in\-context tasks into portable representations: function and task vectors summarize a demonstrated mapping into a single transplantable activation\(Todd et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib31); Hendel et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib13)\), with follow\-ups on their generalization, filter\-like selection heads, concept subspaces, and algorithmic primitives\(Ye et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib39); Sharma et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib26); Tang et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib28); Lippl et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib17)\)\. This literature asks*what*mapping is applied; we ask how a conditional rule is*decomposed*into a predicatetestand an argumentroute, and find that under our probes therouteadmits no separable, transferable subspace\. Transportable filter\-head predicates\(Sharma et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib26)\)independently supporttest\-separability; that setting has no else\-branch or routing analysis, where the asymmetry lies\.
#### Variable binding and routing\.
How models bind entities to roles and route values to slots is central to symbolic\-style computation in transformers\(Feng and Steinhardt,[2024](https://arxiv.org/html/2608.04183#bib.bib7); Wu et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib36)\), with desiderata for probing binding causally\(Davies et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib5)\)and evidence that models mix distinct mechanisms for a single behavior\(Gur\-Arieh et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib11)\)\. Our task is a routing problem in this sense, but the disagreement donors expose, per layer, whether a band holds the condition or the answer word: atest/routestage boundary rather than a single binding operation\.
#### Steering, decodability, and multilingual mechanisms\.
Directional interventions can steer behavior, even conditionally\(Lee et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib16); Arditi et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib1)\), yet a decodable or steerable direction need not be a causally separable, invariant mechanism\(Nadaf,[2026](https://arxiv.org/html/2608.04183#bib.bib22); Opiełka et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib24)\), and interchange success alone is weak evidence: optimization\-found subspaces can pass it while illusory, even on randomly initialized models\(Makelov et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib18); Sutter et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib27)\), and concurrent work stress\-tests head\-level claims by transfer\(Quirke,[2026](https://arxiv.org/html/2608.04183#bib.bib25)\)\. Separately, multilingual analyses find computation routed through a shared, largely English\-centric internal space\(Wendler et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib35); Zhao et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib41)\), with cross\-lingual patching separating concept from language\(Dumas et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib6)\)and near\-identical circuits across two languages\(Ferrando and Costa\-jussà,[2024](https://arxiv.org/html/2608.04183#bib.bib8)\)\. Our transfer control distinguishes a genuine routing subspace from answer\-readout: a learnedroutedirection that flips one label pair but fails to transfer to another is entangled with readout, not separable\. Ourtestband recurs at mid\-stack across six languages and four scripts \(including Tamil and Sinhala, to our knowledge new to mechanistic localization\), and in Gemma\-3\-4B the readout direction for a fixed pair is language\-invariant \(§[4\.2](https://arxiv.org/html/2608.04183#S4.SS2)\)\.
## 3Methodology
### 3\.1Task and Dataset
We study in\-context conditional rules of the form “ifP\(x\)P\(x\)thenAAelseBB\.” The canonical English rule is*“Rule: if the number is greater than 5, output dog, otherwise output cat\.”*Each prompt concatenates the rule, eight in\-context Number/Output demonstrations, and a query number; the model completes a single answer token\. The predicate isP\(x\)≡\(x\>5\)P\(x\)\\\!\\equiv\\\!\(x\>5\), and the answer labelsA=*dog*A=\\text\{\\emph\{dog\}\},B=*cat*B=\\text\{\\emph\{cat\}\}are single tokens\.
All experiments draw from*one*fixed canonical bank of250250items, shared by every experimental cell \(every model and every language\), so no cross\-cell comparison is confounded by a different random sample\. The six languages \(en,zh,hi,id,ta,si\) are rendered from this bank*in full*\(rule and demonstration scaffold alike\), while the numerals and the answer labels*dog*/*cat*stay in Latin script and single\-token form, so cross\-lingual tokenization variation falls on the prompt text but never on the measured answer \(Appendix[A](https://arxiv.org/html/2608.04183#A1)details the translation protocol\): the manipulation targets rule and scaffold comprehension while holding the readout interface constant\. The languages span four scripts and a wide resource range \(Tamil and Sinhala are new to mechanistic localization\)\. Each cell is also rendered under a second single\-token pair \(*fox*/*owl*\); grid curves pool both pairs\. Alternative predicate families \(numeric thresholdst∈\{3,5,7\}t\\\!\\in\\\!\\\{3,5,7\\\}, set membership, vowel\-initial\) live in a separate English\-only bank used for the predicate\-generality experiment\.
### 3\.2Activation Patching and Metrics
We localize computation with activation patching\(Vig et al\.,[2020](https://arxiv.org/html/2608.04183#bib.bib33); Meng et al\.,[2022](https://arxiv.org/html/2608.04183#bib.bib20); Goldowsky\-Dill et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib10)\): run the model on abaseprompt and on a donor prompt, copy the donor’s residual\-stream vector at a single layer, at the last\-token position, into thebaserun, and measure whether the predicted answer changes; repeated per layer, this localizes which band carries which piece of the computation\.
We report two metrics\(Zhang and Nanda,[2024](https://arxiv.org/html/2608.04183#bib.bib40); Heimersheim and Nanda,[2024](https://arxiv.org/html/2608.04183#bib.bib12)\)\. The first is a two\-way*flip probability*over\{A,B\}\\\{A,B\\\}, gated by label mass: a patch counts as a flip only if the combined probability mass on\{A,B\}\\\{A,B\\\}is at least0\.300\.30, so destructive patches that collapse the distribution do not register as flips\. The second is a graded*fraction\-of\-swing*recovery
R=LDpatched−LDbaseLDdonor−LDbase,R\\;=\\;\\frac\{\\mathrm\{LD\}\_\{\\text\{patched\}\}\-\\mathrm\{LD\}\_\{\\text\{base\}\}\}\{\\mathrm\{LD\}\_\{\\text\{donor\}\}\-\\mathrm\{LD\}\_\{\\text\{base\}\}\},whereLD=logit\(A\)−logit\(B\)\\mathrm\{LD\}=\\mathrm\{logit\}\(A\)\-\\mathrm\{logit\}\(B\): how far a patch moves the base logit difference toward the donor’s \(R=0R\\\!=\\\!0no movement,R=1R\\\!=\\\!1full recovery\)\. When this fraction\-of\-swing is computed for a patched head group or subspace rather than a full\-layer residual patch, we write it asΦ\\Phi\. Both flip directions are always run \(predicatetrue→\\toflip andfalse→\\toflip\); Figure[1](https://arxiv.org/html/2608.04183#S3.F1)illustrates the operation\.
Figure 1:The patching operation\. The donor’s last\-token residualhℓh\_\{\\ell\}is copied into thebaserun at layerℓ\\ell\. We then record, at every layer, whether the patched answer flips \(under the label\-mass gate,≥0\.30\\geq 0\.30\) and the graded fraction\-of\-swing recoveryRR\.
### 3\.3The Four Donors
For eachbaseitem \(predicatetrue, answerA=*dog*A=\\text\{\\emph\{dog\}\}\) we construct four prompts by crossing the rule*mapping*\(standard vs\. swapped\) with the query*outcome*\(truevs\.false\), as laid out in Figure[2](https://arxiv.org/html/2608.04183#S3.F2):
- •base: standard rule, querytrue⇒\\Rightarrowpredicatetrue, answer*dog*\(reference run\)\.
- •O\-donor: standard rule, queryfalse⇒\\Rightarrowpredicatefalse, answer*cat*\. Flips the predicate*outcome*; only one query digit changes\.
- •map\-donor: swapped rule \(“…output cat, otherwise dog”\), querytrue⇒\\Rightarrowpredicate staystruebut the answer becomes*cat*\. Flips the*mapping*\.
- •map\-null: swapped rule, queryfalse⇒\\Rightarrowpredicatefalse, answer*dog*\. A large perturbation whose answer nonetheless equalsbase\.
The methodological core is thatmap\-donorandmap\-nullmake the*condition*and the*answer**disagree*\(truth value and answer word point to opposite labels\), letting us read, per layer, a patch’s*allegiance*: whether the patched output follows the truth value \(theteststage\) or the answer word \(theroutestage\)\. The construction is binary here for clarity, but generalizes: a cyclic rotation of the label mapping extends it to a three\-output rule \(§[4\.5](https://arxiv.org/html/2608.04183#S4.SS5)\)\.
Figure 2:The four donors, crossing rule mapping \(standard vs\. swapped\) with query outcome \(truevs\.false\)\. Inmap\-donorandmap\-nullthe predicate truth and the answer word*disagree*, exposing per layer whether a patched site follows the condition \(test\) or the answer \(route\)\.
### 3\.4Controls
A*behavioral gate*admits a cell only if base accuracy is at least0\.850\.85, so that the model actually solves the rule before we interpret its internals\.*Donor\-correctness gating*discards any item whose donor prompts are not themselves answered correctly\. A*perturbation\-matched2×22\\times 2isolation*test contrastsO\-donor\(outcome\-flipping, one digit changed\) againstmap\-donor\(mapping\-flipping, predicate held\), certifying that atest\-band patch reroutes the answer by moving the*condition*rather than by any incidental surface perturbation\.
To rule out a subspace illusion, an*illusion\-guard*compares recovery along the learned truth direction against a random direction of equal norm \(values in §[4\.1](https://arxiv.org/html/2608.04183#S4.SS1)\)\. For the argumentroutewe add a*cross\-label\-pair transfer control*, applying a subspace learned on*dog*/*cat*to a held\-out label pair; failure to transfer marks the router as entangled with readout rather than separable\. We further verify a valid*coherent null*\(null\_cross: the residual of an unrelated base item\) and confirm that an ungated vector null \(null\_vec: a norm\-matched random vector\) collapses once the label\-mass gate is applied, so apparent flips are not artifacts of distributional collapse\. Significance is assessed with McNemar’s test on paired flip outcomes, each cell reporting a single run over its full gated item set\.
#### Verdict criteria\.
All thresholds were fixed before the grid was run: behavioral gate at accuracy≥0\.85\\geq 0\.85; analysis restricted to items whose four donor prompts are all answered correctly;testlayer taken as the argmax of theO\-donorflip curve; and strict single\-layer isolation requiring, at that layer,map\-nullflip≥0\.6\\geq 0\.6withmap\-donorflip≤0\.2\\leq 0\.2\. Per\-cell values appear in Table[1](https://arxiv.org/html/2608.04183#S4.T1), full curves in Appendix[F](https://arxiv.org/html/2608.04183#A6)\.
## 4Experiments and Results
Appendix Figure[12](https://arxiv.org/html/2608.04183#A5.F12)summarizes the design:*predicate\-testlocalization*\(§[4\.1](https://arxiv.org/html/2608.04183#S4.SS1)\) in every cell of a66\-language×\\times33\-model grid over the shared bank; the*router analysis*\(§[4\.2](https://arxiv.org/html/2608.04183#S4.SS2)\) on all three models; generality across families and languages \(§[4\.3](https://arxiv.org/html/2608.04183#S4.SS3)\); and*predicate generality*\(§[4\.4](https://arxiv.org/html/2608.04183#S4.SS4)\) on the primary model\. A recruit\-vs\-build contrast with the weight\-stored IOI circuit is deferred to Appendix[C](https://arxiv.org/html/2608.04183#A3)\. A language enters an analysis only if the model’s base accuracy on the shared bank is at least0\.850\.85\(the behavioral gate\); every cell reported below clears it \(\[0\.926,1\.000\]\[0\.926,1\.000\]\)\.
### 4\.1Localizing the Predicate Test
For each item we run per\-layer residual patching of the four donors \(base,O\-donor,map\-donor,map\-null\) at the last\-token position\. Becausemap\-donorandmap\-nullmake condition and answer*disagree*, the patched prediction reveals, per layer, whether that layer carries the condition \(test\) or the routed answer \(route\)\. The primary metric is the22\-way flip probability under the label\-mass gate, complemented by the graded recoveryRR\(§[3\.2](https://arxiv.org/html/2608.04183#S3.SS2)\)\. Every curve pools two single\-token answer pairs \(*dog*/*cat*,*fox*/*owl*\) and both flip directions\.
Figure 3:Thetest→\\rightarrowroutestage boundary in Gemma\-3\-4B \(English\)\. All patches are applied at the last token of the prompt \(theOutput:position\), which is also the readout site\. The predicate is first computed at the query\-digit position \(L14–15\), read out as a truth value at the mid\-stacktestband \(L21–22, amber\), where theO\-donorreroutes the answer \(1\.001\.00\) but themap\-donordoes not \(0\.000\.00\); patches there follow the*condition*\(map\-donor0\.000\.00,map\-null≈1\.0\\approx 1\.0\)\. At L23 the pattern inverts \(teal\): patches follow the*answer word*, marking the answer\-readoutroutestage\. Gemma\-3\-4B is the*fused*case: readout begins immediately past the band; see Figure[11](https://arxiv.org/html/2608.04183#A5.F11)for the staggered geometries of Gemma\-3\-12B \(readout onset L29–L35, language\-dependent\) and Qwen3\-8B \(L30\)\.#### A separable, isolatedtestmodule\.
Table[1](https://arxiv.org/html/2608.04183#S4.T1)reports the complete 18\-cell grid; every cell but one returnspass\_test\_isolated\(the exception, Qwen3\-8B Tamil, is a layer\-selection artifact; §[4\.3](https://arxiv.org/html/2608.04183#S4.SS3)\): at a single mid\-stack residual band, theO\-donor, which flips only the predicate’s truth value, reroutes the answer with probability≈1\.0\\approx 1\.0, while themap\-donor, which flips the rule’s mapping but leaves the predicatetrue, leaves the answer essentially unchanged \(≈0\.0\\approx 0\.0\)\. The McNemar tests reject the null throughout \(ppbetween1\.4×10−471\.4\\times 10^\{\-47\}and5\.7×10−575\.7\\times 10^\{\-57\}\)\.
Table 1:Predicate\-testlocalization across the 18\-cell grid\. Columns: base accuracy \(all clear the0\.850\.85gate\);test, theO\-donor\-selected layer over model depth \(e\.g\.22/3422/34: layer 22 of 34\);*peak*, the condition peak \(map\-nullargmax\); O/map, theO\-donor/map\-donorflip probabilities; McNemarpp\. Gatednnper cell ranges214214–256256\. Models: Gemma\-3\-4B, Gemma\-3\-12B, Qwen3\-8B\. Every cell but one \(†\) returnspass\_test\_isolated\.†At theO\-donor\-selected layer themap\-nullflip is0\.570\.57, just under the0\.60\.6isolation cutoff: an artifact of the layer rule, not a Tamil effect; at the condition peak it passes, as every Qwen cell does \(§[4\.3](https://arxiv.org/html/2608.04183#S4.SS3)\)\.
#### Stage boundary and controls\.
Figure[3](https://arxiv.org/html/2608.04183#S4.F3)shows the stage structure on the primary cell \(English, Gemma\-3\-4B\)\. Three controls confirm a directed truth\-value edit, not an artifact\. \(i\)*Subspace illusion\-guard*: patching along the learned truth direction recovers the swing at0\.670\.67, versus0\.030\.03for a random direction of equal norm\. \(ii\)*Upstream localization*: patching the query\-digit token position flips the answer most strongly at L14–15 \(0\.830\.83/0\.750\.75\), decaying to zero by L20: the predicate is first computed*at*the number and only later moved to the last token\. \(iii\)*Stage boundary at L22→\\toL23*: within the band \(L20–22\) patches follow the*condition*\(map\-donor0\.000\.00,map\-null0\.970\.97–1\.001\.00\); from L23 the pattern*inverts*\(map\-donor0\.990\.99,map\-null0\.000\.00\), marking the transition into answer readout\. The coherent null \(null\_cross\) stays at≈0\.04\\approx 0\.04, andnull\_vecat≈0\.02\\approx 0\.02once the label\-mass gate is applied\.
### 4\.2Is the Router Separable?
Does theroutestep \(mapping the truth value to the answer word\) have its own separable, transferable subspace? The readout onset \(the first layer where themap\-donorflip reaches0\.90\.9\) differs by model \(Appendix Figure[11](https://arxiv.org/html/2608.04183#A5.F11)\), so each probe is placed at that model’s*own*onset \(Table[2](https://arxiv.org/html/2608.04183#S4.T2)\)\. Method 1 greedily accumulates attention heads over each model’s readout band \(4B L23–30; 12B L32–43; Qwen L30–35\)\. Method 2, “DAS\-lite”, is a low\-rank variant of distributed alignment search\(Geiger et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib9); Wu et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib37)\): a rank\-kkorthogonal basis at one residual site, trained on the interchange objective and scored by interchange intervention accuracy \(IIA\) on held\-out items \(k∈\{1,2,4,8\}k\\in\\\{1,2,4,8\\\},48/3248/32train/test\)\. Both methods run on all three models \(English\); Appendix Figure[14](https://arxiv.org/html/2608.04183#A5.F14)illustrates them on the primary cell, Figure[4](https://arxiv.org/html/2608.04183#S4.F4)the transfer pattern, Table[2](https://arxiv.org/html/2608.04183#S4.T2)all three models\. These probes span greedy head accumulation\(Conmy et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib4)\)over each full readout band and rank\-11–88subspaces at the onset site, tested cross\-pair in every model, cross\-lingually in Gemma\-3\-4B, and under*joint*two\-pair training; a subspace anywhere in this space that transferred across label pairs would overturn the claim, and none does\. The decisive joint case \(Gemma\-3\-4B\): a subspace trained ondog/catandblock/allowtogether fits both atIIA=1\.00\\mathrm\{IIA\}=1\.00down to rank11yet transfers at≤0\.01\\leq 0\.01to held\-outfox/owl; its only partial transfer \(≤0\.46\\leq 0\.46, rank\-unstable\) is toflag/approve, lexically parallel to a trained pair: token geometry, not routing \(Appendix[A](https://arxiv.org/html/2608.04183#A1)\)\.
#### Greedy head\-group patching\.
Where this probe recovers the routing effect at all, themap\-donor\(mapping\) andO\-donor\(outcome\) donors recruit the*same*heads in nearly the same greedy order: identical for the first five of six heads in Gemma\-3\-4B \(L23H3, L23H1, L30H6, L29H4, L29H5;0\.8×0\.8\\timesof the full\-band swing with33heads, full\-bandΦ=0\.96\\Phi=0\.96\), and identical for*all six*in Qwen3\-8B \(L33H9, L35H26, L31H6, L30H5, …; full\-bandΦ=0\.58\\Phi=0\.58\)\. Gemma\-3\-12B differs informatively: patching*all*band heads at once is nearly null \(full\-bandΦ=0\.09\\Phi=0\.09\), yet the greedy\-selected ten\-head group recoversΦ=0\.58\\Phi=0\.58with flips0\.910\.91, led by the same head \(L41H11\) for both donors, nine of ten heads shared\. In no model do distinct routing heads emerge: mapping and outcome share the same machinery, their accumulation curves nearly coincident \(Appendix Figure[10](https://arxiv.org/html/2608.04183#A5.F10)\)\.
#### A learned subspace with a transfer control\.
At each model’s readout layer, a learned low\-rank subspace \(DAS\-lite\) flipsdog↔\\leftrightarrowcatwith interchange accuracyIIA=1\.00\\mathrm\{IIA\}=1\.00on held\-out items at every rankk∈\{1,2,4,8\}k\\in\\\{1,2,4,8\\\}\(Table[2](https://arxiv.org/html/2608.04183#S4.T2)\); yet under the decisive*cross\-label\-pair transfer control*, the same subspace applied to a fresh label pair \(fox/owl\) flips at≤0\.013\\leq 0\.013in every model \(0\.000\.00for Gemma\-3\-12B and Qwen3\-8B,n≥462n\\geq 462\)\. The subspace’s alignment with the answer\-readout direction \(the normalized meanmap\-donor−\-baseresidual difference; largest\|cos\|\|\\cos\|over learned columns\), however,*differs*by family: in the Gemma models it is moderate \(cosine0\.150\.15–0\.390\.39\), so the flip direction is largely the readout direction itself; in Qwen3\-8B it is near zero \(cosine0\.010\.01–0\.090\.09\), a*pair\-specific*direction that neither transfers nor coincides with readout\. Optimization\-found subspaces can pass interchange tests while illusory\(Makelov et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib18); Sutter et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib27)\); the transfer control is the direct antidote\.
Figure 4:The router transfer pattern \(k=1k\{=\}1\)\. In every model the learned subspace flips the trained pair perfectly \(teal\) but transfers to a new pair at≈0\{\\approx\}0\(red\); for Gemma\-3\-4B the same subspace transfers near\-perfectly to the*same*pair in Tamil and Chinese \(light teal\): token\-bound, language\-invariant\.
#### Cross\-lingual transfer: pair\-specific, not language\-specific\.
A second transfer probe pins down what the direction*is*: we apply the English\-learneddog/catsubspace \(Gemma\-3\-4B, L23\),*unchanged*, to the same pair in another language \(only the rule text differs; the labels are shared Latin tokens\)\. Whereas transfer to a new label pair fails \(≤0\.011\\leq 0\.011\), transfer to Tamil and Chinese*succeeds*almost perfectly at every rank \(Tamil0\.970\.97–1\.001\.00, Chinese0\.990\.99; the greedy probe re\-run in this setting recruits the same heads\)\. The router direction is thus*token\-specific but language\-invariant*: the answer\-readout direction for those particular tokens, shared across languages because the tokens are; that token\-binding, not any language binding, makes it non\-transferable\.
Table 2:Router analysis across three models spanning two families \(English\)\. DAS flips the answer perfectly at every rank \(IIAtest=1\.00\\mathrm\{IIA\}\_\{\\text\{test\}\}=1\.00\) but*transfers to a new label pair at≈0\\approx 0*in every model: no separable, transferable router\. Cosine with the readout direction is moderate in Gemma \(the flip is largely readout\) but near\-zero in Qwen \(a pair\-specific direction\)\. The greedy head\-patch recovers the swing in 4B/Qwen but not 12B \(full\-bandΦ\\Phi\), so the 12B row rests on the subspace method alone\.IIA\\mathrm\{IIA\}is on3232held\-out items; transfer is overn≥462n\\geq 462gated items\. A larger\-sample rerun of the primary cell \(150150train,100100test\) givesIIAtest=0\.98\\mathrm\{IIA\}\_\{\\text\{test\}\}=0\.98–0\.990\.99across ranks with transfer≤0\.011\\leq 0\.011, so the interchange result is not a small\-sample effect\.
### 4\.3Generality Across Model Families and Languages
Table[1](https://arxiv.org/html/2608.04183#S4.T1)is itself the cross\-cell evidence against three worries\.*Model idiosyncrasy*: the decomposition replicates across Gemma\-3\-4B, Gemma\-3\-12B\(Team et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib29)\), and Qwen3\-8B\(Yang et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib38)\)\(the cross\-*family*comparison is the real test\), with thetestband at comparable relative depth in each \(Gemma\-3\-4B L21–22,∼0\.63\\sim\\\!0\.63; Gemma\-3\-12B L27–28,∼0\.57\\sim\\\!0\.57; Qwen3\-8B L23–27\)\.*Lexical specificity*: beyond the two pooled pairs of §[4\.1](https://arxiv.org/html/2608.04183#S4.SS1), English reruns with policy\-*action*pairs \(*block*/*allow*,*flag*/*approve*\) and with*multi\-token*answers \(sub\-word splits and two\-word phrases; sequence\-margin metrics\) reproduce the band, isolation, and boundary \(Appendix[B](https://arxiv.org/html/2608.04183#A2)\); the band tracks the predicate’s truth value, not a particular answer word or token count\.*Language specificity*: across six languages spanning four scripts, thetestband sits at the same mid\-stack depth with the same isolation signature \(O\-flip≈1\.0\\approx 1\.0, map\-flip≈0\.0\\approx 0\.0\)\. Both sides are language\-agnostic in different senses: the predicate test localizes at a shared depth, and the readout*direction*is language\-invariant \(the cross\-lingual probe above transfers at≈0\.98\\approx 0\.98\); only the readout*onset*can shift by language \(12B, Figure[11](https://arxiv.org/html/2608.04183#A5.F11)\), consistent with reports of shared mid\-stack multilingual computation\(Wendler et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib35); Zhao et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib41); Tezuka and Inoue,[2025](https://arxiv.org/html/2608.04183#bib.bib30)\)\.
#### A cross\-family difference in band structure\.
The per\-layer curves expose three per\-cell quantities \(Appendix Figure[11](https://arxiv.org/html/2608.04183#A5.F11)\): the*condition peak*\(map\-nullargmax\), the selectedtestlayer \(O\-donorsite\), and the*readout onset*\(§[4\.2](https://arxiv.org/html/2608.04183#S4.SS2)\)\. Gemma\-3\-4B is*fused*: all three sit within two layers in every language \(onset L23 in all six\)\. Qwen3\-8B has*two language\-invariant anchors*: condition peak L23 and readout onset L30 in*all six*languages, the selectedtestlayer varying between them\. This exposes a limit of our pre\-registered layer rule: theO\-donorflips condition*and*answer, so its curve saturates across the corridor between condition peak and readout onset \(Qwen≥0\.95\\geq\\\!0\.95from L23 on\) and the argmax is settled by single\-item differences \(Table[1](https://arxiv.org/html/2608.04183#S4.T1),testvs\. peak\)\. Where the argmax lands past L23,map\-nullhas attenuated and Tamil dips below the0\.60\.6cutoff; at the condition peak*all six*meet the criterion\. The pre\-registered verdicts in Table[1](https://arxiv.org/html/2608.04183#S4.T1)stand\. Gemma\-3\-12B anchors the condition at L27 everywhere but its*readout onset is language\-dependent*\(zh L29; en, hi, ta L32; id, si L35\)\. The same attenuation marks every corridor of two or more layers \(corridor\-meanmap\-nullflips of0\.000\.00–0\.550\.55\) while the matched\-ruleO\-donorstill flips, a graded condition→\\toreadout transition rather than a single boundary\. Throughout, thetestanchor is language\-invariant; only*where*the readout crystallizes varies, not its content \(§[4\.2](https://arxiv.org/html/2608.04183#S4.SS2)\)\.
### 4\.4Predicate Generality
Is the mechanism specific to greater\-than\-55, or a general predicate\-test? We hold the language fixed \(English\) and vary the predicate family, keeping the same format anddog/catlabels so only the predicate changes: numeric thresholdst∈\{3,5,7\}t\{\\in\}\\\{3,5,7\\\}, set\-membershipx∈\{2,3,5,7\}x\\\!\\in\\\!\\\{2,3,5,7\\\}\(a non\-monotonic operator\), and vowel\-initial \(a lexical predicate on letters, not numbers\)\. For each family we localize thetestwith the per\-layer outcome\-donor patch against a coherent null\. Appendix Table[4](https://arxiv.org/html/2608.04183#A5.T4)shows that*every*family exhibits a clean mid\-stacktestband with peak outcome\-flip1\.001\.00and near\-zero null, including the non\-monotonic set\-membership and the non\-numeric vowel predicate; a Qwen3\-8B spot\-check on the two hardest families \(set\-membership, vowel\) reproduces this \(peak\-flip1\.001\.00at L28 and L30\)\. The mechanism is therefore a*predicate\-test*, not a greater\-than\-55circuit \(per\-layer curves in Appendix Figure[13](https://arxiv.org/html/2608.04183#A5.F13)\)\. Selectedtestlayers vary within the band \(L20–L29\), but every family reaches flip≥0\.97\\geq 0\.97from L21 on, including the two whose accuracy sits near the behavioral gate\.
### 4\.5Beyond Binary Rules
We ran a*three\-output*variant on Gemma\-3\-4B \(English\): three numeric buckets map to three single\-token labels \(*dog*/*cat*/*cow*\), and the four donors generalize by a cyclic rotation of the label mapping in place of the binary swap \(construction in Appendix[D](https://arxiv.org/html/2608.04183#A4)\)\. The result matches the binary case: the model solves the rule \(behavioral accuracy0\.910\.91\) and the*same*mid\-stacktestband reappears \(L18–22, stage boundary L22→\\toL23\), with the same isolation signature:O\-donorandmap\-nullflip \(1\.001\.00/0\.980\.98\) whilemap\-donordoes not \(0\.000\.00\) \(Appendix Figure[9](https://arxiv.org/html/2608.04183#A4.F9)\): the predicate\-testmodule is not tied to a two\-label task\.
## 5Discussion
Our central finding is an asymmetry: a single mid\-stack band carries the rule’s truth value and can be overwritten in isolation, across three models and six languages, while under our probes theroutehas no such handle; its learned direction flips only the trained pair\.
Theteststage behaves like a computation over an abstract truth value: the rule text sets up a predicate, evaluated at the query digit and moved to the last token, its result carried on a direction stable enough to swap between prompts\. Because outputs follow the truth value below the boundary and the answer above it, the disagreement design licenses calling thetesta module rather than a shortcut\. Two controls argue the band carries the evaluated truth value rather than a relocated query digit \(the digit’s causal weight decays to zero by L20; a learned truth direction alone recovers0\.670\.67of the swing\), though a cross\-threshold patch, not run here, would settle it\. The condition anchor, moreover, is language\-invariant in every family \(Appendix Figure[11](https://arxiv.org/html/2608.04183#A5.F11)\), as expected of a language\-agnostic truth value; only the readout’s*timing*varies by family and, in 12B, by language\.
Theroute, though, shows no separable, transferable handle\. In all three models a DAS\-lite subspace attains IIA1\.001\.00yet fails to transfer to a new label pair \(≈0\\approx 0; Table[2](https://arxiv.org/html/2608.04183#S4.T2)\)\. What that direction*is*varies by family: in Gemma it aligns moderately with answer readout, the apparent router largely the readout seen through one label pair; in Qwen it is orthogonal to readout, a pair\-specific direction; the invariant is non\-transferability\. The cross\-lingual probe pins the binding: failing on a new pair yet transferring across languages, the direction tracks the answer*tokens*, not the rule’s language or a routing role\. This matches reports of directions that steer effectively without being a faithful, transferable code\(Nadaf,[2026](https://arxiv.org/html/2608.04183#bib.bib22); Opiełka et al\.,[2026](https://arxiv.org/html/2608.04183#bib.bib24)\)\.
Practically, the asymmetry predicts that predicate\-level steering should be more portable across prompts and label sets than answer\-level steering\. In the settings where such conditionals ship \(Appendix[B](https://arxiv.org/html/2608.04183#A2)\), single\-direction steering\(Lee et al\.,[2025](https://arxiv.org/html/2608.04183#bib.bib16); Arditi et al\.,[2024](https://arxiv.org/html/2608.04183#bib.bib1)\)may therefore generalize unevenly, surviving a change of*language*but not*vocabulary*; the transfer test cheaply tells the two apart\. A separabletestalso offers a monitoring hook: the judged condition is readable mid\-stack*before*the action is emitted, a causal target for truth\-probing and judgment/action\-mismatch detection\(Burns et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib2); Marks and Tegmark,[2024](https://arxiv.org/html/2608.04183#bib.bib19)\)\.
Holding the answer tokens fixed in Latin script is a deliberate control: native\-script labels would make the readout select*and*render the answer, confounding routing with translation\.
## 6Conclusion
In\-context conditional execution factors asymmetrically\. The predicatetestis a separable mid\-stack module carrying the rule’s truth value: localized by disagreement donors, isolated under matched controls, stable across three models and six languages \(on Gemma\-3\-4B also five predicates, a three\-output rule, and multi\-token answers\)\. No probe, single\-pair or joint, found a transferableroutesubspace: the direction is answer\-token\-bound, failing new label pairs while transferring across languages in Gemma\-3\-4B\.
## Limitations
Our results come with clear boundaries\. All experiments use three open\-weight models \(Gemma\-3\-4B/12B, Qwen3\-8B\); we cannot patch proprietary systems, so we cannot say whether thetest/routeasymmetry persists at frontier scale\. The task is a deliberately controlled synthetic conditional with*dog*/*cat*labels held in Latin script across languages: this is what makes the flip metrics interpretable and the readout direction shareable across languages\. A multi\-token rerun on the primary cell reproduces the localization under sequence\-margin metrics, but native\-script conditionals and free\-form completions remain open\. Activation patching localizes causally load\-bearing sites; our claims concern localization and separability, not a complete circuit\-level mechanism\. On the router side, the full\-band head patch is nearly null in Gemma\-3\-12B \(Φ=0\.09\\Phi=0\.09; a greedy ten\-head subset recovers0\.580\.58\), so the 12B transfer negative rests on the subspace method; the full two\-method analysis is English\-only, and the cross\-lingual transfer probe was run on Gemma\-3\-4B only \(per\-language subspace training was not repeated\); “entangled with readout” is precise for Gemma while Qwen’s direction is pair\-specific without aligning to readout\. Joint two\-pair training \(§[4\.2](https://arxiv.org/html/2608.04183#S4.SS2)\) closes the single\-pair loophole on the primary model, but wider multi\-pair banks, other sites, and nonlinear alignments remain unprobed\. Lexical generality rests on two label pairs in the grid \(plus two policy\-action pairs and two multi\-token pairs on the primary cell\), the three\-output variant was run on Gemma\-3\-4B in English only, and each cell reports a single seed\. The IOI contrast \(Appendix[C](https://arxiv.org/html/2608.04183#A3)\) is suggestive rather than decisive, since the IOI movers are themselves weak and distributed in Gemma\-3\-4B\. Finally, the six languages sample script and resource level, not typological space\.
## Ethics Statement
This work is a mechanistic\-interpretability study of open\-weight language models \(Gemma\-3\-4B, Gemma\-3\-12B, and Qwen3\-8B\) on a synthetic task\. It involves no human subjects, no user data, and no private or personally identifiable information: every stimulus is generated deterministically from a fixed numeric item bank and a rule template\. The multilingual stimuli are machine\-rendered from the same template into six languages, with the rule and demonstration scaffold translated and the single\-token answer labels held in Latin; the translated rule text was checked by native speakers for each language to ensure the intended predicate is faithfully expressed\. All models are publicly released, and our interventions \(activation patching on the residual stream\) modify only internal activations at inference time, leaving model weights unchanged\.
We note a dual\-use consideration\. Localizing a causally separable predicate\-testmodule identifies a residual\-stream direction at which flipping the truth value reroutes the answer, and such localization could in principle inform activation\-steering attempts to manipulate a model’s conditional behavior\. We regard the risk as limited: our methods require white\-box access to internal activations, the studied task is a controlled synthetic conditional rather than a safety\-relevant behavior, and the same interpretability tools are what make such manipulations auditable and detectable\. We report these findings to advance scientific understanding of how in\-context rules are assembled into runtime circuits, and we encourage that steering insights derived from interpretability be applied toward transparency and control rather than covert behavioral manipulation\.
## References
- Arditi et al\. \(2024\)Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda\. 2024\.Refusal in language models is mediated by a single direction\.*Advances in Neural Information Processing Systems*, 37:136037–136083\.
- Burns et al\. \(2023\)Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt\. 2023\.Discovering latent knowledge in language models without supervision\.In*The Eleventh International Conference on Learning Representations*\.
- Chen et al\. \(2026\)Danchun Chen, Qiyao Yan, and Liangming Pan\. 2026\.Towards a mechanistic understanding of propositional logical reasoning in large language models\.*arXiv preprint arXiv:2601\.04260*\.
- Conmy et al\. \(2023\)Arthur Conmy, Augustine Mavor\-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga\-Alonso\. 2023\.Towards automated circuit discovery for mechanistic interpretability\.*Advances in Neural Information Processing Systems*, 36:16318–16352\.
- Davies et al\. \(2023\)Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau\. 2023\.Discovering variable binding circuitry with desiderata\.*arXiv preprint arXiv:2307\.03637*\.
- Dumas et al\. \(2025\)Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West\. 2025\.[Separating tongue from thought: Activation patching reveals language\-agnostic concept representations in Transformers](https://doi.org/10.18653/v1/2025.acl-long.1536)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 31822–31841\.
- Feng and Steinhardt \(2024\)Jiahai Feng and Jacob Steinhardt\. 2024\.How do language models bind entities in context?In*International Conference on Learning Representations*, volume 2024, pages 36391–36413\.
- Ferrando and Costa\-jussà \(2024\)Javier Ferrando and Marta R\. Costa\-jussà\. 2024\.[On the similarity of circuits across languages: a case study on the subject\-verb agreement task](https://doi.org/10.18653/v1/2024.findings-emnlp.591)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 10115–10125\.
- Geiger et al\. \(2024\)Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman\. 2024\.Finding alignments between interpretable causal variables and distributed neural representations\.In*Causal Learning and Reasoning*, pages 160–187\. PMLR\.
- Goldowsky\-Dill et al\. \(2023\)Nicholas Goldowsky\-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora\. 2023\.Localizing model behavior with path patching\.*arXiv preprint arXiv:2304\.05969*\.
- Gur\-Arieh et al\. \(2026\)Yoav Gur\-Arieh, Mor Geva, and Atticus Geiger\. 2026\.Mixing mechanisms: How language models retrieve bound entities in\-context\.In*The Fourteenth International Conference on Learning Representations*\.
- Heimersheim and Nanda \(2024\)Stefan Heimersheim and Neel Nanda\. 2024\.How to use and interpret activation patching\.*arXiv preprint arXiv:2404\.15255*\.
- Hendel et al\. \(2023\)Roee Hendel, Mor Geva, and Amir Globerson\. 2023\.In\-context learning creates task vectors\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 9318–9333\.
- Hong et al\. \(2025\)Guan Zhe Hong, Nishanth Dikkala, Enming Luo, Cyrus Rashtchian, Xin Wang, and Rina Panigrahy\. 2025\.A implies B: Circuit analysis in LLMs for propositional logical reasoning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Kim et al\. \(2025\)Geonhee Kim, Marco Valentino, and Andre Freitas\. 2025\.[Reasoning circuits in language models: A mechanistic interpretation of syllogistic inference](https://doi.org/10.18653/v1/2025.findings-acl.525)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 10074–10095\.
- Lee et al\. \(2025\)Bruce W Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar\. 2025\.Programming refusal with conditional activation steering\.In*International conference on learning representations*, volume 2025, pages 90960–90985\.
- Lippl et al\. \(2026\)Samuel Lippl, Thomas McGee, Kimberly Lopez, Ziwen Pan, Pierce Zhang, Salma Ziadi, Oliver Eberle, and Ida Momennejad\. 2026\.Algorithmic primitives and compositional geometry of reasoning in language models\.In*Proceedings of the 43rd International Conference on Machine Learning*\.
- Makelov et al\. \(2024\)Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda\. 2024\.Is this the subspace you are looking for? An interpretability illusion for subspace activation patching\.In*The Twelfth International Conference on Learning Representations*\.
- Marks and Tegmark \(2024\)Samuel Marks and Max Tegmark\. 2024\.The geometry of truth: Emergent linear structure in large language model representations of true/false datasets\.In*First Conference on Language Modeling*\.
- Meng et al\. \(2022\)Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\. 2022\.Locating and editing factual associations in GPT\.*Advances in neural information processing systems*, 35:17359–17372\.
- Merullo et al\. \(2024\)Jack Merullo, Carsten Eickhoff, and Ellie Pavlick\. 2024\.Circuit component reuse across tasks in transformer language models\.In*International Conference on Learning Representations*, volume 2024, pages 18349–18377\.
- Nadaf \(2026\)Mohammed Suhail B Nadaf\. 2026\.Steerable but not decodable: Function vectors operate beyond the logit lens\.*arXiv preprint arXiv:2604\.02608*\.
- Olsson et al\. \(2022\)Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al\. 2022\.In\-context learning and induction heads\.*arXiv preprint arXiv:2209\.11895*\.
- Opiełka et al\. \(2026\)Gustaw Opiełka, Hannes Rosenbusch, and Claire E Stevenson\. 2026\.Causality≠\\neqinvariance: Function and concept vectors in LLMs\.In*The Fourteenth International Conference on Learning Representations*\.
- Quirke \(2026\)Philip Quirke\. 2026\.Ablation\-reversible heads don’t transfer: A stress test for mechanistic role claims in Transformers\.*arXiv preprint arXiv:2606\.08292*\.
- Sharma et al\. \(2026\)Arnab Sen Sharma, Giordano Rogers, Natalie Shapira, and David Bau\. 2026\.LLMs process lists with general filter heads\.In*The Fourteenth International Conference on Learning Representations*\.
- Sutter et al\. \(2025\)Denis Sutter, Julian Minder, Thomas Hofmann, and Tiago Pimentel\. 2025\.The non\-linear representation dilemma: Is causal abstraction enough for mechanistic interpretability?In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Tang et al\. \(2026\)Wei Tang, Xinyan Jiang, Fakhri Karray, and Lijie Hu\. 2026\.In\-context learning operates as concept subspace learning\.*arXiv preprint arXiv:2605\.18830*\.
- Team et al\. \(2025\)Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 192 others\. 2025\.[Gemma 3 technical report](https://arxiv.org/abs/2503.19786)\.*Preprint*, arXiv:2503\.19786\.
- Tezuka and Inoue \(2025\)Hinata Tezuka and Naoya Inoue\. 2025\.The transfer neurons hypothesis: An underlying mechanism for language latent space transitions in multilingual LLMs\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 31742–31792\.
- Todd et al\. \(2024\)Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau\. 2024\.Function vectors in large language models\.In*International conference on learning representations*, volume 2024, pages 17282–17333\.
- Variengien and Winsor \(2025\)Alexandre Variengien and Eric Winsor\. 2025\.Look before you leap: Universal emergent mechanism for retrieval in language models\.In*The Thirteenth International Conference on Learning Representations \(ICLR\)*\.
- Vig et al\. \(2020\)Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber\. 2020\.Investigating gender bias in language models using causal mediation analysis\.*Advances in neural information processing systems*, 33:12388–12401\.
- Wang et al\. \(2023\)Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt\. 2023\.Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.In*The Eleventh International Conference on Learning Representations*\.
- Wendler et al\. \(2024\)Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West\. 2024\.Do Llamas work in English? On the latent language of multilingual transformers\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15366–15394\.
- Wu et al\. \(2025\)Yiwei Wu, Atticus Geiger, and Raphaël Millière\. 2025\.How do transformers learn variable binding in symbolic programs?In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 67284–67299\. PMLR\.
- Wu et al\. \(2023\)Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman\. 2023\.Interpretability at scale: Identifying causal mechanisms in Alpaca\.*Advances in neural information processing systems*, 36:78205–78226\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\.
- Ye et al\. \(2025\)Qinyuan Ye, Robin Jia, and Xiang Ren\. 2025\.Function induction and task generalization: An interpretability study with off\-by\-one addition\.*arXiv preprint arXiv:2507\.09875*\.
- Zhang and Nanda \(2024\)Fred Zhang and Neel Nanda\. 2024\.Towards best practices of activation patching in language models: Metrics and methods\.In*The Twelfth International Conference on Learning Representations*\.
- Zhao et al\. \(2024\)Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing\. 2024\.How do large language models handle multilingualism?*Advances in Neural Information Processing Systems*, 37:15296–15319\.
## Appendix AReproducibility Details
#### Prompt template\.
Each prompt is a rule, eight in\-context Number/Output demonstrations, and a query number whose answer token the model completes\. The canonical English rule is “Rule: if the number is greater than55, output dog, otherwise output cat\.” Answer labels*dog*/*cat*are single tokens and stay in Latin script across all six languages \(en, zh, hi, id, ta, si\); the rule and the entire demonstration scaffold are translated, so tokenization variation falls on the prompt text and never on the measured answer\. Figure[5](https://arxiv.org/html/2608.04183#A1.F5)shows one English prompt per predicate family \(each abbreviated to two of eight demonstrations\)\. The multilingual banks were built by translating this template \(rule text and scaffold keywords\) into each of the other five languages and having the result checked and corrected by a native speaker, keeping the numerals and the answer tokens unchanged\.
Figure 5:English prompts for each predicate family \(abbreviated to two of eight demonstrations\)\. Only the predicate changes; the task format and the single\-tokendog/catlabels are held fixed\. Non\-English banks translate the rule and scaffold keywords, with native\-speaker correction, leaving numerals and answer tokens unchanged\.
#### Fixed item bank\.
Every experimental cell \(each model×\\timeseach language\) draws from one shared bank of250250items with the numeric greater\-than predicate at threshold55; the shared bank ensures no cell is confounded by different random samples\. Predicate families for the generality analysis live in a separate English\-only bank; both banks ship with the code release\.
#### Four\-donor construction\.
The four donors cross rule mapping \(standard/swapped\) with query outcome \(true/false\) exactly as defined in §[3\.3](https://arxiv.org/html/2608.04183#S3.SS3); we do not repeat them here\.
#### Models and precision\.
Gemma\-3\-4B \(3434layers\), Gemma\-3\-12B \(4848layers\), and Qwen3\-8B \(3636layers\), all run inbfloat16\. Layers are0\-indexed over decoder blocks; a patch at layerℓ\\ellcopies the donor’s residual\-stream vector at the block\-ℓ\\elloutput, at the last token position, into the base run\.
#### Metrics\.
After patching one layer we score the completion\. The flip probability is a22\-way mass over\{A,B\}\\\{A,B\\\}with a label\-mass gate: a patch counts only when the combined label mass is≥0\.30\\geq 0\.30, so destructive or incoherent patches do not register as flips\. The graded fraction\-of\-swing recovery is
R=LDpatched−LDbaseLDdonor−LDbase,R=\\frac\{\\mathrm\{LD\}\_\{\\text\{patched\}\}\-\\mathrm\{LD\}\_\{\\text\{base\}\}\}\{\\mathrm\{LD\}\_\{\\text\{donor\}\}\-\\mathrm\{LD\}\_\{\\text\{base\}\}\},withLD=logit\(A\)−logit\(B\)\\mathrm\{LD\}=\\mathrm\{logit\}\(A\)\-\\mathrm\{logit\}\(B\)\. Both flip directions \(true→\\rightarrowflip andfalse→\\rightarrowflip\) are run\.
#### DAS\-lite training\.
The subspace is a semi\-orthogonal basisQ∈ℝd×kQ\\in\\mathbb\{R\}^\{d\\times k\}\(orthogonal parametrization\), trained with Adam \(learning rate10−210^\{\-2\}, batch1616,150150epochs\) to minimize the cross\-entropy of the patched logits against the counterfactual mapping target; IIA is argmax\-among\-labels agreement with that target on held\-out items, and the null is an untrained random subspace of matched rank\. Every experiment reports a single seed per cell\.
#### Joint two\-pair DAS\.
Single\-pair training cannot in principle discover a router shared across pairs, so on the primary cell \(English, Gemma\-3\-4B\) we retrained the subspace jointly ondog/catandblock/allow\(interleaved batches,2424train/1616test per pair, same site, objective, and gates\)\. At every rankk∈\{1,2,4,8\}k\\in\\\{1,2,4,8\\\}the joint subspace reachesIIAtest=1\.00\\mathrm\{IIA\}\_\{\\text\{test\}\}=1\.00on*both*trained pairs \(rank11suffices for both jointly\), yet transfers at≤0\.009\\leq 0\.009to the unrelated held\-out pairfox/owl\(n=114n\{=\}114; random\-subspace null0\.000\.00\)\. Transfer toflag/approve\(n=112n\{=\}112\) is partial and rank\-unstable \(0\.000\.00,0\.460\.46,0\.070\.07,0\.320\.32fork=1,2,4,8k\{=\}1,2,4,8\): what transfer exists follows the lexical similarity of the answer tokens \(block∼\{\\sim\}flag,allow∼\{\\sim\}approve\) and vanishes for unrelated vocabulary, the signature of readout\-geometry overlap rather than an abstract routing variable\.
#### Grid layout\.
In the released code, results are organized per language, model, and experiment \(one directory per cell\), so the per\-cell tables in the main text map directly onto the release; the experiments there carry short internal identifiers \(E1 = predicate\-testlocalization, E2 = router separability, E3 = predicate generality, E6 = IOI contrast, E7 = the three\-output rule, E8 = multi\-token answers, E9 = joint two\-pair DAS\)\. The cross\-model and cross\-language analyses \(internally E4/E5\) are computed from the stored E1 outputs and have no separate directories\.
#### Compute and software\.
All experiments run on single workstation\-class NVIDIA GPUs inbfloat16via PyTorch and HuggingFacetransformers; no run requires more than one GPU\. A localization cell \(one model×\\timesone language\) takes roughly one to two GPU\-hours on Gemma\-3\-4B and proportionally more for the larger models; the router analysis \(greedy sweep \+ DAS training\) takes one to two GPU\-hours per model, and the three\-output run under one GPU\-hour\. No distributed training is involved; model weights are never updated\. Gemma\-3 models are used under the Gemma Terms of Use and Qwen3\-8B under its Apache\-2\.0 license\.
## Appendix BRealistic Conditional Rules
The studied template is the minimal shared skeleton of conditional instructions that ship in real prompts\. Each maps onto the same three roles: a predicate over an input, a true\-branch action, a false\-branch action\.
Table 3:Deployed prompt patterns instantiating “ifP\(x\)P\(x\)thenAAelseBB\.” The third pattern is, up to wording, the studied rule, and is run directly below\.Our canonical rule isolates this skeleton under controlled conditions: single\-token branch labels \(relaxed to multi\-token labels below\) make the flip metrics exact, and a numeric predicate makes truth assignment unambiguous\. What the findings imply for the deployed forms follows the two stages\. Thetestband is separable, language\-invariant, and generalizes across predicate families, label pairs, a three\-output rule, and multi\-token answers; this suggests the*judgment*half of such rules is a stable target for probing and portable steering\. Therouteresult cautions that any intervention built on the*action*half is token\-bound: it will not survive a change of action vocabulary \(*refuse*→\\to*decline*\), though it does survive a change of prompt language\.
As a direct check, we reran the localization on the primary cell \(English, Gemma\-3\-4B\) with two policy\-*action*pairs \(*block*/*allow*and*flag*/*approve*, both single tokens\) in place of the noun pairs, with no other change\. The model solves the rule at accuracy0\.9840\.984\(n=228n\{=\}228gated items\), and the localization reproduces exactly:testlayer L20 withO\-donorflip0\.990\.99,map\-donor0\.000\.00,map\-null0\.940\.94\(McNemarp=3\.3×10−50p=3\.3\\times 10^\{\-50\}\), nulls≤0\.031\\leq 0\.031, the same L22→\\toL23 stage boundary, and the same verdict as the grid cells \(pass\_test\_isolated; Figure[6](https://arxiv.org/html/2608.04183#A2.F6)\)\. The skeleton’s behavior is unchanged when its branches are the action words of the deployed patterns above\.
Figure 6:Policy\-action labels \(Gemma\-3\-4B, English;*block*/*allow*and*flag*/*approve*pooled\): per\-layer flip for theO\-donor\(outcome\),map\-null\(condition\) andmap\-donor\(answer\) donors\. The shaded band \(layers withO\-donorflip≥0\.6\\geq 0\.6andmap\-donorflip≤0\.2\\leq 0\.2\) contains the noun\-labeltestband, and the L22→\\toL23 stage boundary is identical\.A second check removes the single\-token constraint itself\. We generalized the metrics to*multi\-token*answers \(the flip becomes a teacher\-forced sequence\-margin sign change, with the patch held at the prompt’s last\-token position while candidate tokens are scored, and the mass gate applied to the candidates’ first tokens\) and reran the localization with two multi\-token pair types pooled: sub\-word\-split single words \(*wheelbarrow*/*grasshopper*\) and two\-word phrases \(*snow leopard*/*tree frog*\), each two tokens with distinct first tokens\. The model solves the rule at accuracy0\.9640\.964\(n=168n\{=\}168gated\), and the localization reproduces once more:testlayer L20 \(condition peak L21\),O\-donorflip0\.990\.99,map\-donor0\.020\.02,map\-null0\.940\.94\(McNemarp=6\.8×10−37p=6\.8\\times 10^\{\-37\}\), nulls≤0\.04\\leq 0\.04, and the same L22→\\toL23 boundary \(Figure[7](https://arxiv.org/html/2608.04183#A2.F7)\)\. Gemma\-3’s256256k vocabulary makes virtually every common English noun a single token, so multi\-token answers require sub\-word\-split or multi\-word candidates by construction\.
Figure 7:Multi\-token answers \(Gemma\-3\-4B, English; sub\-word pair and two\-word\-phrase pair pooled, sequence\-margin metrics\): per\-layer flip for theO\-donor\(outcome\),map\-null\(condition\) andmap\-donor\(answer\) donors\. The shaded band \(same criterion as Figure[6](https://arxiv.org/html/2608.04183#A2.F6)\) contains the single\-tokentestband, with the same L22→\\toL23 boundary\.Untested here, and required before these implications carry to production rules: native\-script actions, compositional predicates, and free\-form rather than constrained\-choice completions \(§[4\.3](https://arxiv.org/html/2608.04183#S4.SS3); Limitations\)\.
## Appendix CRecruit vs\. Build: The IOI Contrast
Is the conditional circuit*recruited*from a pretrained mechanism or*built*at inference? We contrast it \(English, Gemma\-3\-4B\) with the canonical weight\-stored circuit, Indirect Object Identification \(IOI\)\(Wang et al\.,[2023](https://arxiv.org/html/2608.04183#bib.bib34)\): “When John and Mary went to the store, John gave a drink to”→\\to“Mary”\. With the same interchange machinery, we rank the IOI*mover*heads \(those carrying the IO/S logit difference to the last token under a role\-swap corruption\) and measure their overlap with our conditional’s heads\.
Two caveats frame one clean result \(Figure[8](https://arxiv.org/html/2608.04183#A3.F8)\)\. First, the IOI movers are individually*weak*here \(the strongest recovers onlyΦ=0\.27\\Phi=0\.27of the swing, three of ten with zero OV copy\-score\), so IOI in Gemma\-3\-4B is itself distributed and the mover ranking is a weak\-signal set; the contrast is suggestive, not decisive\. Second, since mover heads live late by construction, zero mid\-stack overlap is partly expected; the*depth\-matched*comparison is against therouteband, and there the asymmetry appears:*no*IOI mover falls in the predicate\-testband L20–22 \(overlap0\.000\.00\), while the depth\-matchedrouteheads share33of1010movers \(L23H3, L26H3, L29H4; overlap0\.300\.30, Jaccard0\.180\.18\), consistent with theroutedrawing on general late\-layer readout machinery that IOI also uses, rather than a dedicated recruited circuit\. Reported per head\-set rather than collapsed to one number, the verdict is*test: distinct \(built\) /route: shared*: the sametest/routeasymmetry the router analysis found \(§[4\.2](https://arxiv.org/html/2608.04183#S4.SS2)\), seen from a different angle\.
Figure 8:IOI\-contrast head map \(Gemma\-3\-4B\): IOI mover heads \(black, sized byΦ\\Phi\) never fall in thetestband \(amber\); three coincide with ourrouteheads \(teal circles; shared heads ringed red\)\.
## Appendix DThe Three\-Output Rule
The three\-output variant \(§[4\.5](https://arxiv.org/html/2608.04183#S4.SS5)\) uses three numeric buckets with two thresholds \(x≥5x\\geq 5,2≤x≤42\\leq x\\leq 4,x≤1x\\leq 1\) mapped to three single\-token labelsA,B,CA,B,C\(*dog*/*cat*/*cow*\)\. In place of the binary swap, the mapping is a cyclic rotationr∈\{0,1,2\}r\\in\\\{0,1,2\\\}: bucketbbtakes label\(b\+r\)mod3\(b\{\+\}r\)\\bmod 3, so the rule always contains the three label words once each and every prompt tokenizes to the same length\. Demonstrations cover all three buckets\. For a base query in the top bucket under the standard mapping \(answerAA\), with common flip targetBB, the four donors are:base\(r=0r\{=\}0, top bucket→A\\to A\);O\-donor\(r=0r\{=\}0, middle bucket→B\\to B, a one\-boundary move\);map\-donor\(r=1r\{=\}1, top bucket→B\\to B, rotated map,*same*bucket\); andmap\-null\(r=2r\{=\}2, middle bucket→A\\to A, rotated map whose answer returns to the base\)\. A condition\-carrying layer follows the donor’s bucket: the base’s own rule then maps the middle bucket toBB, soO\-donorandmap\-nullflip whilemap\-donor\(same bucket\) does not; a readout layer follows the answer word, somap\-donorandO\-donorflip whilemap\-nulldoes not\. This is the binary isolation logic with a rotation standing in for the swap\. Figure[9](https://arxiv.org/html/2608.04183#A4.F9)shows the resulting per\-layer curves\.
Figure 9:Three\-output rule \(Gemma\-3\-4B\): per\-layer flip for theO\-donor\(outcome\),map\-null\(condition\) andmap\-donor\(answer\) donors of the*dog*/*cat*/*cow*variant\. The shaded band \(L18–22, same criterion as Figure[6](https://arxiv.org/html/2608.04183#A2.F6)\) contains the binarytestband, and the L22→\\toL23 stage boundary is identical: below it patches follow the condition \(O\-donor,map\-null\), above it the answer word \(map\-donor\)\.
## Appendix EAdditional Figures
Figures[10](https://arxiv.org/html/2608.04183#A5.F10)–[14](https://arxiv.org/html/2608.04183#A5.F14)provide the greedy head\-accumulation curves, the per\-cell band geometry, the predicate\-generality curves, the experimental\-design overview, and the router\-methods schematic referenced from the main text\.
Figure 10:Greedy head\-group patching: fraction\-of\-swingΦ\\Phias heads are added in greedy order, for the mapping and outcome donors \(English\)\. The two curves are nearly coincident in every model: the same heads, recruited in nearly the same order, carry both the mapping and the outcome, so no distinct routing heads emerge\. Dotted line: patching the*whole*band at once\. Gemma\-3\-4B reaches0\.8×0\.8\\timesthe full\-band effect with three heads; in Gemma\-3\-12B the greedy group far exceeds the full\-band patch \(0\.580\.58vs\.0\.090\.09\), i\.e\. patching all band heads together largely cancels\.Figure 11:Condition peak \(map\-nullargmax, amber square\), selectedtestlayer \(O\-donorsite, open teal circle\), and readout onset \(firstmap\-donorflip≥0\.9\\geq 0\.9, filled teal dot\) for every cell, computed from the stored per\-layer curves\. Gemma\-3\-4B fuses all three; Qwen3\-8B holds*both*the condition \(L23\) and the readout onset \(L30\) language\-invariant with thetestsite between them; Gemma\-3\-12B anchors the condition \(L27\) but its readout onset varies by language \(L29–L35\)\.Figure 12:Experimental design\. One fixed 250\-item bank feeds every cell of the66\-language×\\times33\-model localization grid \(only the rule text is translated; the single\-token Latin labels are held fixed\)\. The router analysis runs on the English cells of all three models; the teal outline marks the primary cell \(English, Gemma\-3\-4B\), which also hosts the cross\-lingual transfer probe and the label\-generality replications\. Predicate generality and the IOI contrast are English\-only companion analyses\.Figure 13:Predicate generality \(Gemma\-3\-4B\): per\-layerO\-donorflip for all five predicate families, each reaching1\.01\.0in the mid\-stack band; triangles mark selectedtestlayers\. Bottom strip: Qwen3\-8B spot\-check \(set\-membership, vowel\)\. Max null flip0\.0620\.062\.Figure 14:Two probes of the argumentroutestep, shown on the primary cell \(Gemma\-3\-4B, English; see Table[2](https://arxiv.org/html/2608.04183#S4.T2)for all three models\)\.*Left:*greedy head\-group patching over L23–30 recruits the same heads in the same order for the mapping \(map\-donor\) and the outcome \(O\-donor\) donors\.*Right:*a DAS\-lite subspace at L23 flips dog↔\\leftrightarrowcat \(IIA=1\.00\\text\{IIA\}=1\.00\) but does not transfer to a new label pair \(fox/owl,≈0\.01\\approx 0\.01; cosine with the answer\-readout direction0\.270\.27–0\.390\.39\)\. Verdict:entangled\_with\_readout\.Table 4:Predicate generality \(Gemma\-3\-4B, English\)\. Every predicate family localizes a clean mid\-stacktestband with peak outcome\-flip1\.001\.00and null≤0\.05\\leq 0\.05, including non\-monotonic set\-membership and the non\-numeric vowel predicate\. A Qwen3\-8B spot\-check reproduces set\-membership \(L28\) and vowel \(L30\) at peak\-flip1\.001\.00\.
## Appendix FPer\-Language Flip Curves
Figures[15](https://arxiv.org/html/2608.04183#A6.F15)–[17](https://arxiv.org/html/2608.04183#A6.F17)plot the per\-layer flip probabilities behind Table[1](https://arxiv.org/html/2608.04183#S4.T1): one panel pair per model, six languages overlaid, with theO\-donorpatch left, themap\-donorpatch right, and shading marking the selectedtestlayers\. They show the per\-language coincidence of thetestband, the stage boundary, and the readout\-onset geometry of §[4\.3](https://arxiv.org/html/2608.04183#S4.SS3)\.
Figure 15:Gemma\-3\-4B: per\-layer flip probability for all six languages\. Left:O\-donor; right:map\-donor\. Shading: range of selectedtestlayers \(L21–22\)\.Figure 16:Gemma\-3\-12B: as above;testband L27–28\. Themap\-donorrise varies by language downstream of the boundary while theO\-donorcurves coincide\.Figure 17:Qwen3\-8B: as above;testband L23–27\. Themap\-donorrise is displaced several layers past the band: the staggered condition→\\toreadout structure\.Similar Articles
CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment
CounterRoute introduces an online reinforcement-learning framework that jointly learns routing and mode-conditioned responses in dual-mode language models, improving accuracy while reducing inference tokens.
Belief-reality separation lives in routing over a shared value slot in language models
This paper investigates how language models separate a character's belief from reality, finding that they use a shared value slot for attributed values and a router at the query position to select the frame (belief or reality) to read out. It identifies two routes for asserted and derived beliefs, and shows that the slot itself carries no belief-reality tag; the separation lies in dissociated routing subspaces.
Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing
Introduces Local Branch Routing (LBR), a token-level test-time scaling framework that expands a local lookahead tree and uses a lightweight router to select the best branch. LBR improves reasoning on mathematical benchmarks over chain-of-thought and other baselines.
A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
This paper investigates whether bilingual Mixture-of-Experts (MoE) language models develop linguistically structured expert routing. It finds that interpretable linguistic organization emerges within MoE routing patterns, and that curriculum training influences specialization in language balance.
Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models
This paper uses causal interventions to investigate syntactic mechanisms in multilingual language models, revealing cross-lingual transfer that is graded based on typological similarity.