Planning or Improvisation? Stress-Testing the Poetry Planning Site on Open Models and Open Cross-Layer Transcoders

# Planning or Improvisation? Stress-Testing the Poetry Planning Siteon Open Models and Open Cross-Layer Transcoders
Source: [https://arxiv.org/html/2609.18440](https://arxiv.org/html/2609.18440)
###### Abstract

[Lindsey et al\. \(2025\)](https://arxiv.org/html/2609.18440#bib.bib10)report that Claude 3\.5 Haiku plans rhymes: features for candidate rhyme words are active on the newline before a line is written, and a suppress\-and\-inject intervention redirects the line only when applied there \(their Figure 13\)\. We test how far this generalizes on seven cells crossing four open models \(0\.6B to 2\.6B parameters\) with six open cross\-layer transcoders \(CLTs\), on one consumer GPU, decomposing the claim into position specificity \(C1\), newline site identity \(C2\), and a newline\-resident plan \(C3\)\. This is a stress test rather than a faithful reproduction: attribution graphs are unavailable for these CLTs, so features are found bottom\-up from decoder vectors\. C1 generalizes, in every cell and in all 247 of 444 prompt\-by\-inject pairs with a detectable effect, but the effective position is the final prompt token, adjacent to emission, and only two cells reach behaviorally meaningful probabilities\. C2 and C3 are not recovered by any probe: a census of every active feature finds no rhyme\-anticipating enrichment at the newline, and steering the newline while the model composes the whole line, over 36 runs and 8,640 sampled lines, shows why\. That intervention is strong but one token long, making the injected word the first word of the composed line in 703 of 720 samples and leaving the rhyme six words later untouched\. A final test drops the transcoder entirely: patching the newline’s whole residual, at every layer, from a minimal\-pair poem whose third line ends on a different rhyme moves the rhyme in 11 of 1,260 composed lines against 4 at baseline, with a design resolving 1\.4%\. We read this as a boundary condition rather than a refutation: at this scale and with these transcoders, the causal site is emission\-adjacent\. We reproduce Figure 13’s shape, not its mechanism\. Code and data are public\([Jacopin, 2026b](https://arxiv.org/html/2609.18440#bib.bib7)\)\.

## 1Introduction

Whether language models plan ahead is a central question for interpretability, and rhyming poetry is its cleanest published test case\.[Lindsey et al\. \(2025\)](https://arxiv.org/html/2609.18440#bib.bib10)distinguish two mechanisms by which a model could end a line with a rhyme:*pure improvisation*, choosing the word only when it must be emitted, and*planning*, committing to a candidate ending before the line is written and composing toward it\. For Claude 3\.5 Haiku they report evidence for planning, and the sharpest piece of that evidence is a steering\-location experiment \(their Figure 13\): suppressing the planned\-word features and injecting an alternative changes the completion*only*when the intervention is applied at the newline token preceding the line, several tokens upstream of the emitted word\.

Figure 13 is an attractive target for a generalizability study\. It is causal rather than correlational, it makes a falsifiable point prediction \(effect concentrated at one specific, upstream position\), and it was produced with a proprietary model and a proprietary 30M\-feature cross\-layer transcoder \(CLT\), so it has not been checked end to end outside Anthropic\. Recent open\-model work approaches it from different angles:[Hanna and Ameisen \(2026\)](https://arxiv.org/html/2609.18440#bib.bib4)reproduce rhyme planning on Qwen3 models with CLTs and argue that latent planning emerges with scale, and[Maar et al\. \(2026\)](https://arxiv.org/html/2609.18440#bib.bib12)shift rhyming behavior with contrastive steering vectors\.[Jacopin \(2026d\)](https://arxiv.org/html/2609.18440#bib.bib9)reports position\-specific CLT steering on Gemma 2 2B and Llama 3\.2 1B, but at a site that differs from Anthropic’s, a discrepancy that prior work has not analyzed and that we make explicit and central here\.

#### What this paper is and is not\.

This is a stress test of the generalizability of Figure 13, in the sense of the reproducibility track it was written for: the same intervention machinery, applied beyond the original’s experimental scope \(open models two to three orders of magnitude smaller than Claude 3\.5 Haiku, open CLTs ten to two thousand times narrower\)\. It is*not*a faithful reproduction of the original method\. The original discovers planned\-word features top\-down, by attribution from a specific completion; no attribution\-graph tooling exists for these open CLTs at consumer scale, so we discover features bottom\-up, from their decoder vectors\. This difference is load\-bearing and we return to it repeatedly: bottom\-up discovery finds features that*write*a rhyme word, and can only test whether a plan is visible through such features\. We state, for each claim, what our probes can and cannot show \(§[2](https://arxiv.org/html/2609.18440#S2), §[7](https://arxiv.org/html/2609.18440#S7)\)\.

#### Contributions\.

Following the track’s themes, we testgeneralizability\(across open models and CLTs\),ablation\(which protocol components carry the effect\),baselines\(non\-CLT steering, per\-layer transcoders, and the random controls of the dead\-salmon lineage,[Méloux et al\., 2025](https://arxiv.org/html/2609.18440#bib.bib13)\), andevaluation\(which metric choices change the conclusion\)\. Our answers, from seven \(model, CLT\) cells swept over intervention position and strength, plus a re\-analysis of 444 prompt\-by\-inject pairs:

1\. Position specificity \(C1\) generalizes, but is partly generic\.In 7/7 cells the intervention is effective at exactly one position, and in 247/247 pairs with a detectable effect that position is the final prompt token \(§[4\.1](https://arxiv.org/html/2609.18440#S4.SS1)\)\. Random write\-directions also concentrate their leverage there, at far smaller magnitudes: the*location*of the spike is a property of the emission\-adjacent write path, its*magnitude*is what distinguishes the rhyme features \(§[4\.2](https://arxiv.org/html/2609.18440#S4.SS2)\)\.2\. Behavioral redirection in two of seven cells\.Only the two mntss cells reach probabilities that change generated text \(0\.48, 0\.85\); the five Qwen3 cells move probabilities of10−710^\{\-7\}to10−310^\{\-3\}, detectable but not behavioral \(§[4\.1](https://arxiv.org/html/2609.18440#S4.SS1)\)\.3\. The newline site \(C2\) and a newline plan \(C3\) are not recovered\.The newline is inert for the rhyme target in every cell and every pair and carries no enrichment of rhyme\-anticipating features \(§[4\.3](https://arxiv.org/html/2609.18440#S4.SS3)\)\. Steered while the model composes the whole line, it captures the next token and nothing beyond it: the injected word opens the line in 1,540 of the 1,545 lines it appears in, while the rhyme is never redirected under any of six counting criteria \(§[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)\)\.4\. Magnitude belongs to the transcoder, not the model\(six orders of magnitude across CLTs, §[4\.6](https://arxiv.org/html/2609.18440#S4.SS6)\)\.5\. The protocol’s free parameters matter:the original documents no steering strength; the optimum varies froms=1s\{=\}1tos=25s\{=\}25across cells, and swapping one inject feature moves a cell from3\.5×3\.5\\timesto33,860×33\{,\}860\\times\(§[5](https://arxiv.org/html/2609.18440#S5)\)\.

All experiments run on one consumer GPU111An RTX 5060 Ti \(16 GB\), Rust/candle stack\([HuggingFace, 2023](https://arxiv.org/html/2609.18440#bib.bib5);[Jacopin, 2026a](https://arxiv.org/html/2609.18440#bib.bib6)\)\. Commands, artifacts, runtimes, and feature identifiers are in Appendix[B](https://arxiv.org/html/2609.18440#A2)\.; every number regenerates from committed artifacts\.

## 2The Original Result and the Claims Under Test

[Lindsey et al\. \(2025\)](https://arxiv.org/html/2609.18440#bib.bib10)prompt Claude 3\.5 Haiku with a couplet opening \(“He saw a carrot and had to grab it,”\) and study how the model completes the second line with “rabbit”\. Attribution graphs computed from a 30M\-feature CLT localize the decision on the newline token before the second line: features for the candidate words “rabbit” and “habit” are active there, and, per their caption, “the features are only active over the planning token”\. Their Figure 13 then sweeps the position of a suppress\-and\-inject intervention \(suppress the “rabbit” and “habit” features, inject a “green” feature found in a separate poem\) across the prompt and plots the probability of predicting a different final token: “the steering intervention only has an effect over the newline planning token”\. In a separate test on a random sample of 25 poems, injecting a planned\-word feature made the model end its line with the injected word in 70% of cases\. Two of their own caveats matter for what follows: no numerical steering strength is documented, and planned\-word features were found “in about half of the poems”, which the authors attribute either to incomplete CLT coverage or to the model not always planning\.

#### Three claims\.

We decompose the result into three separately testable claims\.C1 \(position specificity\):the intervention effect is concentrated at a single position; other positions are inert\.C2 \(site identity\):that position is the newline preceding the line, upstream of the emitted word, rather than the position adjacent to emission\.C3 \(plan representation\):features representing the planned word are active at the planning site before the line is written\. C1 without C2 and C3 is compatible with improvisation: an intervention adjacent to emission that directly biases the output distribution\. C2 and C3 are what make the result evidence of*planning*\. The distinction matters:[Jacopin \(2026d\)](https://arxiv.org/html/2609.18440#bib.bib9)describes its position sweeps as replicating Figure 13, but its effective site is the final prompt token, so it confirms C1 while, as we show below, C2 and C3 are not recovered\.

Table 1:Protocol differences between the original steering\-location experiment and this stress test\. The feature\-discovery row is the one that limits what our probes can show \(§[7](https://arxiv.org/html/2609.18440#S7)\)\.
#### What our probes can and cannot show\.

Table[1](https://arxiv.org/html/2609.18440#S2.T1)lists the protocol differences\. Two are fixed throughout\. First, bottom\-up feature discovery biases discovered features toward output\-adjacent \(“sayXX”\) behavior\. A steering test built on such features can show that*these*write\-directions do not redirect from the newline; it cannot show that no newline\-resident plan exists, because a plan represented in features that our scan never surfaces would be invisible to it\. The census of §[4\.3](https://arxiv.org/html/2609.18440#S4.SS3)is designed to reduce this gap \(it classifies every active feature, not only scan hits\), but its plan\-like criterion still reads decoders\. Second, our primary sweeps use prompts that end immediately before the rhyme word; §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)runs the composition\-horizon variant that matches the original’s geometry\. Throughout, a claim we “do not recover” is one that none of our probes finds; we reserve “refute” for nothing in this paper\.

## 3Experimental Setup

#### Models and transcoders\.

We use four open base models\([Gemma Team, 2024](https://arxiv.org/html/2609.18440#bib.bib3);[Llama Team, 2024](https://arxiv.org/html/2609.18440#bib.bib11);[Qwen Team, 2025](https://arxiv.org/html/2609.18440#bib.bib15)\), spanning two sizes of the same family \(Qwen3\-0\.6B and 1\.7B\) and two architecture families \(Llama 3\.2 1B, Gemma 2 2B\), crossed with six open CLTs from two independent pipelines: themntssplain\-ReLU CLTs \(426K to 2\.5M features\) and the BlueLightAI JumpReLU CLTs \(16K and 20K per layer\)\([mntss, 2025](https://arxiv.org/html/2609.18440#bib.bib14);[BlueLightAI, 2026](https://arxiv.org/html/2609.18440#bib.bib1)\)\. Crossing models, CLTs, and rhyme families yields the seven cells of Table[3](https://arxiv.org/html/2609.18440#S4.T3)\. CLTs are the cross\-layer descendant of sparse\-autoencoder dictionary learning\([Templeton et al\., 2024](https://arxiv.org/html/2609.18440#bib.bib17)\)\. The CLT implementation is validated against the Python Circuit Tracer reference\([Safety Research, 2025](https://arxiv.org/html/2609.18440#bib.bib16)\): 90/90 top\-10 features match with maximum relative error1\.2×10−61\.2\\times 10^\{\-6\}\.

#### Feature discovery\.

For each CLT we scan every feature’s decoder vector against the full token\-embedding matrix, keep features whose top tokens are clean English words, transcribe them with the CMU Pronouncing Dictionary\([Carnegie Mellon University, 2014](https://arxiv.org/html/2609.18440#bib.bib2)\), and group them by rhyme ending \(last stressed vowel onward\)\. This yields 287 phonologically clean features for Gemma 426K \(35 rhyme groups, 98 words\), 79 for Llama 524K, and 636 to 739 for the Qwen3 CLTs\.

#### Intervention protocol\.

Following the original’s suppress\-and\-inject design: all discovered features of the prompt’s natural rhyme group are clamped at strength−s\-s, one feature of an alternative group is added at\+s\+s, both applied from the feature’s source layer through all downstream layers, at a single token position\. We sweep the position over every prompt token and the strength overs∈\{0\.5,1,2\.5,5,10,25,50,100\}s\\in\\\{0\.5,1,2\.5,5,10,25,50,100\\\}\(a 2D grid\), and measure the next\-token probability of the injected word\. Prompts are four\-line completion poems ending in a trailing space \(Appendix[A](https://arxiv.org/html/2609.18440#A1)\); base models require this priming to rhyme at all \(a 0% rhyme rate on bare prompts rises to 78% with priming\)\.

#### Effect\-size tiers\.

Because ratios over tiny baselines make weak effects look strong, every steering result is reported with its absolute probability and assigned to one of three tiers:*behavioral*\(P≥0\.1P\\geq 0\.1, the injected word competes for the greedy output\),*marginal*\(10−3≤P<0\.110^\{\-3\}\\leq P<0\.1\), and*logit\-only*\(P<10−3P<10^\{\-3\}, detectable in the distribution but with no effect on sampled text at these temperatures\)\.

Table 2:Sample sizes per experiment\. Single\-prompt experiments are marked as such in the text; Limitations discusses why the prompt universe is small at this scale\.

## 4Results

### 4\.1C1: one effective position, behavioral in two cells of seven

Table[3](https://arxiv.org/html/2609.18440#S4.T3)summarizes the seven cells; Figure[1](https://arxiv.org/html/2609.18440#S4.F1)shows four position sweeps\. In every cell the sweep is flat at baseline for every position except one, where it rises by a factor of3\.83\.8to10710^\{7\}depending on the cell\. Six of seven cells spike at the final prompt token; the seventh spikes one position earlier\. The reference cells replicate and slightly exceed[Jacopin \(2026d\)](https://arxiv.org/html/2609.18440#bib.bib9): Gemma0\.4820\.482ats=25s\{=\}25against0\.4570\.457at that work’s fixeds=10s\{=\}10; Llama0\.8530\.853against0\.7770\.777\.

The tiers of Table[3](https://arxiv.org/html/2609.18440#S4.T3)qualify the headline\. Only the two mntss cells are behavioral: the injected word reaches0\.480\.48and0\.850\.85, enough to change generated text\. The Qwen3\-0\.6B cell with the 16K development CLT is marginal \(0\.0090\.009\), and the four Qwen3 cells with the production 20K CLTs are logit\-only, moving the target from about10−710^\{\-7\}to at most4\.5×10−64\.5\\times 10^\{\-6\}\. The single\-position*signature*is present in all seven;*redirection*in the sense of the original’s 70% is present in two\.

#### Breadth\.

Two checks address the single prompt per cell\. With each reference cell’s intervention held fixed, the three other validated prompts of the earlier study, including prompts whose natural rime is foreign to the injected feature, all spike at the final token \(8/8 prompts; Clopper\-Pearson 95% CI\[0\.40,1\]\[0\.40,1\]per cell; joint uniform\-position null7\.2×10−137\.2\\times 10^\{\-13\}\)\. And we re\-analyzed the committed position sweeps of[Jacopin \(2026c\)](https://arxiv.org/html/2609.18440#bib.bib8), run at a fixeds=10s\{=\}10over four prompts and every alternative rhyme group discovered for each CLT: 136 pairs on Gemma 426K, 44 on Llama 524K, 264 on Gemma 2\.5M \(Table[8](https://arxiv.org/html/2609.18440#A3.T8)\)\. Of the 247 pairs whose best position exceeds ten times the sweep’s own floor, 247 spike at the final token; all 28 pairs with a behavioral\-tier effect do\. The pairs whose maximum falls on a newline \(29 of 444\) are all flat profiles whose maximum is within2\.6×2\.6\\timesof the floor, that is, noise\. Under a uniform\-position null for the seven grid cells \(each spike equally likely at any of itsni∈\[20,32\]n\_\{i\}\\in\[20,32\]positions\), all seven localizing within their last two positions has probability∏i\(2/ni\)=3\.3×10−8\\prod\_\{i\}\(2/n\_\{i\}\)=3\.3\\times 10^\{\-8\}\(Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\.

Table 3:The seven cells: models\([Gemma Team, 2024](https://arxiv.org/html/2609.18440#bib.bib3);[Llama Team, 2024](https://arxiv.org/html/2609.18440#bib.bib11);[Qwen Team, 2025](https://arxiv.org/html/2609.18440#bib.bib15)\)×\\timesCLTs\([mntss, 2025](https://arxiv.org/html/2609.18440#bib.bib14);[BlueLightAI, 2026](https://arxiv.org/html/2609.18440#bib.bib1)\)×\\timesrhyme families\. Best \(strength, position\) per cell from the full 2D grid; ratio is bestPPover baselinePPof the injected word; tiers as defined in §[3](https://arxiv.org/html/2609.18440#S3)\. BLA is BlueLightAI\. Every cell spikes at the final prompt token except†\(one position before\)\. In all seven cells the suppress and inject features are encoder\-silent at the final prompt token \(§[4\.3](https://arxiv.org/html/2609.18440#S4.SS3)\), so at the effective site every cell tests bidirectional write\-direction steering rather than suppression of an active feature\. The word\-level mntss 2\.5M CLT \(ReLU\), used in §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)and Table[8](https://arxiv.org/html/2609.18440#A3.T8), completes the open\-CLT inventory\.Figure 1:Position sweeps for four cells \(log scale\), at each cell’s best strength\. Orange bars are newline tokens; the blue bar is the spike\. The dashed line is the unsteered baseline probability of the injected word\. The intervention is inert at every newline and effective only at the final prompt token, adjacent to emission\. Note the vertical scales: the two top panels reach behavioral probabilities, the two bottom panels do not\.

### 4\.2Is the final\-token spike generic?

A reviewer of an earlier version of this paper raised the right objection: the injected features are decoder\-aligned, so injecting them at the final position gives a direct path to the next\-token logits, whereas an earlier intervention must propagate through attention\. The sweep may therefore identify the position of maximal causal leverage rather than the position at which the model commits to the rhyme\. We test this directly with the per\-position random controls \(Figure[3](https://arxiv.org/html/2609.18440#A3.F3), Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\.

On the three cells above the logit\-only tier, we replaced the inject feature with 10 layer\-matched random CLT features or 10 norm\-matched Gaussian directions \(suppress side and strength unchanged\) and recorded, at every position, the target’s probability and, for random features, the probability of the drawn feature’s own top decoder token\. The location*is*generic: random features spike their own token at the final position in 10/10 \(Gemma\), 9/10 \(Llama\), and 6/10 \(Qwen\) draws, by ratios of up to10610^\{6\}over their own tiny baselines; the remaining draws spike nowhere\. The magnitude is not: the largest own\-token probability any random draw reaches is3\.6×10−63\.6\\times 10^\{\-6\}\(Gemma\),2\.9×10−32\.9\\times 10^\{\-3\}\(Llama\),2\.3×10−32\.3\\times 10^\{\-3\}\(Qwen\), against0\.480\.48,0\.850\.85, and0\.0090\.009for the real feature\. On the two behavioral cells the real feature exceeds the best random write\-direction by two to five orders of magnitude; on the marginal Qwen cell it exceeds it by4×4\\timesonly, which is why we do not count that cell as evidence of feature\-specific redirection\. No random draw moves the*target*above7×10−57\\times 10^\{\-5\}at any position\.

Two further controls bound the triviality\. For a feature whose source layer is the final layer, a final\-token spike is architecturally forced \(no attention follows; the logit readout sees only the last position\)\. But the effect persists for features injected at middle layers: the strongest Gemma \-out feature \(layer 16 of 26, nine downstream attention layers\) still concentrates its effect at the final token \(160,379×160\{,\}379\\timesover baseline under inject\-only steering ats=10s\{=\}10, though to an absolute probability of only3\.6×10−43\.6\\times 10^\{\-4\}\)\. And on a Gemma built from config with seeded random weights, or with every trained tensor’s elements shuffled in place, the identical pipeline and features produce no spike \(worst ratios3\.5×3\.5\\timesand99×99\\timesagainst9,974,880×9\{,\}974\{,\}880\\timestrained; §[5](https://arxiv.org/html/2609.18440#S5)\)\.

We therefore restate C1 more carefully than the original’s framing invites\. The single\-position signature says where a write\-direction has the most leverage, and at this scale that is the position adjacent to emission\. What is informative about planning is not the spike but its complement: the newline, which in Claude 3\.5 Haiku was the*only*effective position, is here inert \(§[4\.3](https://arxiv.org/html/2609.18440#S4.SS3)\), including when the model has a whole line to compose \(§[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)\)\.

### 4\.3C2 and C3: the newline is not recovered as a site

Every prompt contains three newline tokens, including the one immediately preceding the incomplete final line, the exact analogue of the original’s planning site\. Across all seven cells and all eight strengths, no newline position moves the target probability off baseline \(Gemma 426K:P≈4\.8×10−8P\\approx 4\.8\\times 10^\{\-8\}at positions 9, 17, and 25, against0\.4570\.457at the final token under the identicals=10s\{=\}10intervention\)\. The pair\-level re\-analysis extends this to 444 pairs: the best newline position never exceeds1\.4×1\.4\\times\(Gemma 426K\),3\.3×3\.3\\times\(Llama\), or1\.3×1\.3\\times\(Gemma 2\.5M\) the sweep’s floor \(Table[8](https://arxiv.org/html/2609.18440#A3.T8)\)\. The failure is not for want of causal reach: interventions at the newline have 6 to 25 subsequent token positions and, for mid\-layer features, up to 9 downstream attention layers through which to propagate\.

Steering with the*wrong*features could still explain the inert newline: as §[2](https://arxiv.org/html/2609.18440#S2)fixes, our discovery yields “sayXX” features by construction, while the original’s attribution surfaced planned\-word features resident on the newline\. We therefore ran a census: at each newline, at mid\-line control tokens, and at the final token, encode the residual through the CLT and classify*every*active feature \(not a top\-KK, so weak features cannot be truncated away\) by whether its decoder points at the upcoming rhyme \(two or more rhyme\-group words in its top\-20 vocabulary projection, or cosine≥0\.3\\geq 0\.3to the target word\)\. The result is uniform across all seven cells: the plan\-like rate at the newline never exceeds the mid\-line base rate \(Gemma: 0\.00% at newlines against 0\.28% at controls; Llama: 1\.02% against 2\.06%; the dense JumpReLU codes sit flat at 0\.03% to 0\.76% everywhere\)\. Rhyme\-decoding features fire at a constant background rate; nothing concentrates at the newline\. So C3 is not recovered either: there is no CLT\-visible representation of the upcoming rhyme at the newline to steer \(full tables in Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\. The census is one prompt per cell, and its criterion reads decoders, so a plan carried by features whose decoders do not surface rhyme words would evade it; see §[7](https://arxiv.org/html/2609.18440#S7)\.

#### At the effective site, every cell is write\-direction steering\.

The census also records the encoder activation of the protocol’s own steering features at each censused position \(Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\. In all seven cells, the suppress features and the inject feature are encoder\-silent at the final prompt token, the position where the intervention works\. Clamping a silent feature at−s\-ssuppresses nothing; it injects a negative decoder direction\. At the effective site the suppress\-and\-inject intervention is therefore bidirectional write\-direction steering, decoupled from the model’s actual feature activations, which explains mechanically why that site is emission\-adjacent, and which means none of our cells tests the original’s suppression of an*active*planned\-word feature\. It also settles what the suppress half contributes in the inject\-only ablation \(§[5](https://arxiv.org/html/2609.18440#S5)\): a second write\-direction, not the removal of an active feature\. On the mntss cells the steering features are silent at every censused position\. On the JumpReLU Qwen3 cells they are weakly active \(0\.05 to 0\.29\) at some mid\-line control tokens, and on Qwen3\-1\.7B \-ation two suppress features are active at newlines \(15:263 at 0\.11 and 0\.53 on the second and third newlines, 18:4404 at 0\.26 on the first\), which is the one place in our data where a feature for the natural rhyme group is present at the original’s planning site\. The census counts it \(it does not raise that cell’s newline plan\-like rate above control, Table[5](https://arxiv.org/html/2609.18440#A3.T5)\), and the grid shows that clamping it there at any of the eight strengths leaves the target at baseline\.

The census also caught our own reference code manufacturing the opposite conclusion: an earlier detection analysis in the implementation underlying[Jacopin \(2026d\)](https://arxiv.org/html/2609.18440#bib.bib9)read the encoder from the post\-MLP residual and reported rhyme features active at the final token\. Under the hook the CLT was actually trained on, those activations are exactly 0\.000 \(Appendix[D](https://arxiv.org/html/2609.18440#A4)\)\.

### 4\.4The composition\-horizon test

The strongest remaining objection to the sweeps above is that prompts ending immediately before the rhyme word leave the model nothing to plan\. We therefore rerun the experiment in the original’s geometry: truncate the prompt at the newline ending line 3, so the model must compose the entire fourth line, and steer at that newline, which is now also the final prompt token, handing the newline\-time signal its best possible chance\. Per condition \(baseline, suppress\-only, inject\-only, suppress\-and\-inject\) we record sampled lines \(temperature 0\.7\) classified by the CMU rime of their final word, the greedy line, and the teacher\-forced probability of the inject word at the final\-word slot, swept over every steering position \(Figure[2](https://arxiv.org/html/2609.18440#S4.F2)\)\.

Four cells carry this test: the two behavioral cells, the marginal Qwen3 cell, and the word\-level 2\.5M\-feature Gemma CLT, whose features are the closest open analogue to the “ordinary features representing the planned word” of the original\. Because an earlier version of this experiment used one prompt and 20 lines per condition, which cannot separate a near\-zero baseline from anything below 40%, we reran it at the scale its conclusion requires: every validated prompt per cell \(four on the Gemma cells, three on Llama, one on Qwen3\) crossed with three sampling seeds, 60 lines per condition per run, 36 runs and 8,640 sampled lines in total \(Table[2](https://arxiv.org/html/2609.18440#S3.T2)\)\. Decision criteria were registered before the runs and are reproduced in Appendix[C](https://arxiv.org/html/2609.18440#A3); all analysis scripts and per\-line outputs are in the repository\.

Figure 2:Composition\-horizon sweeps \(log scale\), one representative run per cell\. The prompt ends at the line\-3 newline \(orange, arrow\); lighter bars are the composed baseline line;P⁡\(inject word\)P\(\\text\{inject word\}\)is measured at the final\-word slot\. The newline sits at the sweep’s floor for the rhyme target, including at word\-level CLT resolution \(top right\); only steering inside the composed line, adjacent to emission, moves it\. All 36 runs of the experiment agree: the peak falls inside the composed line in 36 of 36 and at the newline in 0 of 36, and the newline stays between 0\.86 and 1\.24 times its own floor\.#### The newline moves the next token, and only the next token\.

The intervention at the newline is not weak; it is short\-ranged\. It captures the token immediately after the steering site: the injected word becomes the*first word*of the composed line in 703 of 720 sampled lines on Gemma 426K under suppress\-and\-inject, against 0 of 720 at baseline, and in 470 of 540 on Llama against 1 of 540 \(Table[6](https://arxiv.org/html/2609.18440#A3.T6)\)\. Of the 1,545 lines across all four cells in which the injected word appears under suppress\-and\-inject, it is the line’s first word in 1,540\. What it does not do is survive to the end of the line\. Six words later, at the rhyme, the same intervention has left no trace\.

This is the emission\-adjacent mechanism of §[4\.2](https://arxiv.org/html/2609.18440#S4.SS2)seen end on\. In this geometry the newline*is*the final prompt token, so steering there biases the next\-token distribution exactly as it does in the primary sweeps; the composed line then proceeds from that word under the model’s own dynamics\. The position sweeps make the same point from the other side\. Across all 36 runs the sweep peaks at a position inside the composed line, never at the newline \(36/36 and 0/36\), and the newline sits at its own floor in every run, at a ratio of 0\.86 to 1\.24 \(median 0\.99 on Gemma 2\.5M, 1\.02 on Gemma 426K, 0\.88 on Llama\)\. The peak reaches a medianP⁡\(inject\)P\(\\text\{inject\}\)of 0\.88 on Gemma 426K, 0\.74 at word level and 0\.999 on Llama, against a newline median of6\.7×10−56\.7\\times 10^\{\-5\},1\.1×10−61\.1\\times 10^\{\-6\}and6\.4×10−46\.4\\times 10^\{\-4\}\.

#### The rhyme is never redirected\.

Under the registered metric, the fraction of composed lines whose final word falls in the injected rhyme group, no cell separates from its baseline: Gemma 426K 38/720 against 21/720, Llama 4/540 against 0/540, Qwen3 2/180 against 0/180, Gemma 2\.5M 0/720 against 0/720, with overlapping intervals everywhere \(Table[6](https://arxiv.org/html/2609.18440#A3.T6)\)\. H1 fails in all four cells\.

That metric is confounded, and in the direction that flatters the planning hypothesis: a line that begins with “around” and ends with “around” or “found” is counted as redirected\. We therefore recount the inject\-group endings under criteria that exclude lines into which the word was inserted, from the registered metric through to “the injected word does not occur in the line at all” \(Table[7](https://arxiv.org/html/2609.18440#A3.T7)\)\. The nominal rise on Gemma 426K, the only cell with any counts to speak of, survives no control: 21 to 38 raw \(p=0\.03p=0\.03\), 17 to 23 once lines ending on the injected word itself are removed \(p=0\.42p=0\.42\), and then it*reverses*, to 20 against 4 \(p=1\.4×10−3p=1\.4\\times 10^\{\-3\}\) and to 20 against 0 under the two strictest criteria \(p=1\.7×10−6p=1\.7\\times 10^\{\-6\}and1\.4×10−51\.4\\times 10^\{\-5\}\)\. Steering does not add genuine redirects; it removes the ones the model produces on its own\. At baseline Gemma 426K ends 20 of 720 lines cleanly in the inject rhyme group, with well\-formed completions such as “Her father was nowhere to be found\.”; under either condition that injects, it ends none\. On the other three cells the count of genuine redirects under steering never exceeds two under any criterion, and under the strictest one it is zero in every cell but for a single Qwen3 line\.

This recount was not pre\-registered and we report it as exploratory; Table[7](https://arxiv.org/html/2609.18440#A3.T7)exists so the reader can apply whichever criterion they consider fair, including the registered one\. The conclusion is the same under all six\.

#### What the design can exclude\.

Overlapping intervals do not show equivalence, so we state the resolution\. Pooled over prompts and seeds, each condition holds 720 lines on the Gemma cells, 540 on Llama and 180 on Qwen3\. The smallest redirect fraction whose interval would clear the baseline’s is 0\.013 on Gemma 2\.5M, 0\.017 on Llama, 0\.050 on Qwen3 and 0\.061 on Gemma 426K, the last because its baseline is a non\-zero 21/720: the prompts’ natural group \(AW1 T, “about”\) and the injected group \(AW1 N D, “around”\) share a nucleus, so the model lands in the injected group spontaneously\. Every cell therefore excludes an effect of the size the original reports, 70% of lines on 25 poems, by a very wide margin, and the three cells with near\-zero baselines exclude anything above about 2%\. What we cannot exclude is a redirect rate below one or two percent\.

#### Composition is derailed, not degraded\.

Two further checks separate the mechanism from two plausible alternatives\. Newline injection does not simply break the model: the share of lines repeating a word three or more times is statistically unchanged on Gemma 426K \(2\.6% to 3\.5%,p=0\.36p=0\.36\) and Llama \(0\.4% to 0\.2%\), and*falls*on Gemma 2\.5M \(2\.6% to 0\.7%,p=6×10−3p=6\\times 10^\{\-3\}\), while the type\-token ratio moves by at most 0\.02 and upward, toward more varied lines, in three cells of four\. Lines do shorten on Gemma 426K, from 7\.4 to 6\.1 words\. Nor is the natural rhyme destroyed: the registered contrasts on the natural\-group rate give one result below the corrected threshold ofα=0\.0042\\alpha=0\.0042, inject\-only on Gemma 426K \(220/720 to 164/720,p=1\.0×10−3p=1\.0\\times 10^\{\-3\}\), whose sign is inconsistent across the four prompts; the drop reported in the earlier version of this paper on Llama does not replicate at scale \(p=0\.048p=0\.048\)\. H2 therefore fails as a claim about plan disruption: what the numbers describe is a line that begins elsewhere, not a plan that has been disturbed\. The typical steered line is “Around the beach he soon found out\.”, which starts on the injected word and still ends in the prompt’s own rhyme group\.

Across three model families, two CLT pipelines, both feature granularities, 36 runs and 8,640 composed lines, the verdict is uniform: steering at the newline rewrites the beginning of the line and leaves its rhyme untouched, while the only position that redirects the rhyme is the one adjacent to emission\. This is consistent with[Hanna and Ameisen \(2026\)](https://arxiv.org/html/2609.18440#bib.bib4), whose smallest models show the weakest planning signal, and with the collapse of rhyme rates under forced non\-default targets that we report in Limitations\.

### 4\.5Patching the newline, without a transcoder

Every probe so far writes a CLT decoder direction, so every null can be blamed on feature discovery \(§[2](https://arxiv.org/html/2609.18440#S2)\)\. Activation patching cannot be: it*replaces*a residual row with one the model itself produced, needing no features, no transcoder and no strength\. We therefore ran it as the decisive test of reading \(ii\)\.

#### Design\.

Donor and recipient are a*minimal pair*: two prompts identical through line 3 except that line’s final word, which sets a different rhyme\. Patching the last prompt position transfers its whole state, so with arbitrary prompts a rhyme change would only show that the recipient inherited the donor’s context; with a minimal pair, everything the patch can carry is shared except the rhyme\. The donor’s row is patched into the recipient’s line\-3 newline, one layer at a time and at every layer at once, and the model then composes line 4\. Seven prompt\-direction pairs on Gemma 2 2B×\\timesthree seeds give 1,260 sampled lines per condition\. Criteria were registered before the runs \(Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\.

Only Gemma carries this test\. The design needs AABB prompts, where line 3 opens a new couplet, and Llama 3\.2 1B rhymes on those in 0 to 8% of lines against Gemma’s 20 to 42%, below the 15% its own baseline would need for a redirect to be detectable\. That is consistent with its validated prompts being monorhyme and with prior reports that Llama rhymes less reliably than Gemma\([Jacopin, 2026c](https://arxiv.org/html/2609.18440#bib.bib8)\)\. The result is therefore single\-model, and we report it as such\.

#### The newline does not carry the rhyme\.

Replacing the newline’s entire state, at every layer, with that of a poem ending on a different rhyme moves the composed line’s rhyme in 11 of 1,260 lines against 4 at baseline \(Fisherp=0\.12p=0\.12\); one layer at a time, 10 of 1,260 \(p=0\.18p=0\.18\)\. The design resolves a redirect rate of 1\.4%\. The recipient’s*own*rhyme is equally untouched, 384/1,260 against 357/1,260 \(p=0\.26p=0\.26\)\. The patch is not inert on the text: the composed line opens with the donor’s line\-3 word 33 times against 4 at baseline \(p=9×10−7p=9\\times 10^\{\-7\}\), the same next\-token capture as §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4), at 2\.6% rather than 98% because a minimal\-pair patch carries so little\.

That "so little" is itself measured, and it is not nothing: the donor’s and recipient’s newline rows differ down to cosine 0\.925\. The newline does encode which word line 3 ended on\. It is what the model does with that difference, namely nothing to the rhyme six words later, that this experiment establishes\.

#### Controls\.

Patching a prompt with its own row is a bit\-exact no\-op in all 42 runs \(maximum probability shift1\.4×10−81\.4\\times 10^\{\-8\}\)\. A donor whose line 3 ends in a CMU rime\-mate of the recipient’s does not move the rhyme \(136/420 to 114/420,p=0\.11p=0\.11\), so the instrument does not move rhymes for reasons unrelated to the plan\. Patching at a mid\-line position instead, where a minimal pair’s rows are identical by construction, is an exact no\-op \(136/420 to 136/420,p=1\.000p=1\.000\); that null is evidence the harness writes what it claims, and an earlier version that failed it exposed a real defect \(Appendix[D](https://arxiv.org/html/2609.18440#A4)\)\.

### 4\.6Magnitude is a property of the transcoder

Holding the protocol fixed, the best redirect ratio varies by more than six orders of magnitude across CLTs \(Table[3](https://arxiv.org/html/2609.18440#S4.T3)\), and the ordering follows the transcoder, not the model\. Both mntss cells sit at10510^\{5\}to107×10^\{7\}\\timeswith absolute probabilities of 0\.48 to 0\.85; the BlueLightAI production cells sit at3\.83\.8to157×157\\timeswith absolute probabilities that never exceed5×10−65\\times 10^\{\-6\}\. The 16K development CLT lands in between \(33,860×33\{,\}860\\times\) for a reason instructive in itself \(§[5](https://arxiv.org/html/2609.18440#S5)\)\. A study that had tested only Qwen3 with the production 20K CLTs would have reported the steering result as marginal; one that had tested only Gemma with mntss would have called it dramatic\. Neither would have been wrong about its own cell\.

The evaluation metric compounds this\. The33,860×33\{,\}860\\timescell has an absolute redirect probability of0\.0090\.009; the157×157\\timescell,4\.5×10−64\.5\\times 10^\{\-6\}\. Ratios over tiny baselines make weak effects look strong, and the original reports neither a ratio nor a baseline, only an intervention\-condition probability\. We recommend reporting absolute probability, baseline, and a behavioral readout, as Table[3](https://arxiv.org/html/2609.18440#S4.T3)and §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)do\.

### 4\.7No monotonic scaling within Qwen3

Within the matched pair \(same CLT pipeline, same width, same prompts, same protocol\), the 0\.6B model shows a*stronger*final\-token spike than its 1\.7B sibling on both rhyme families:157×157\\timesagainst16\.4×16\.4\\times\(\-teen\) and5\.4×5\.4\\timesagainst3\.8×3\.8\\times\(\-ation\), all four in the logit\-only tier\. Feature availability also favors the smaller model \(739 against 636 phonologically clean features\)\. This does not contradict the 0\.6B\-to\-14B trend of[Hanna and Ameisen \(2026\)](https://arxiv.org/html/2609.18440#bib.bib4), which we cannot test above 1\.7B on consumer hardware, but it shows the within\-family trend is not monotonic at its lower end, and that CLT training quality is a plausible confound in any cross\-size comparison\.

## 5Ablations and Controls

#### Steering strength\.

The original documents no numerical steering strength; thes=10s\{=\}10used by[Jacopin \(2026d\)](https://arxiv.org/html/2609.18440#bib.bib9)is that work’s internal convention\. Figure[3](https://arxiv.org/html/2609.18440#A3.F3)\(bottom right, appendix\) shows why this free parameter deserves a grid: the optimum iss=25s\{=\}25on both mntss reference cells,s=5s\{=\}5on Qwen3\-1\.7B \(\-teen\), ands=1s\{=\}1on Qwen3\-0\.6B \(\-teen\), where the response declines monotonically with strength\. Three qualitative regimes appear \(rise\-plateau, rise\-decline, monotone decline\), so a single global strength misstates every cell but one\.

#### Inject\-only versus suppress\-and\-inject\.

Because CLT decoder vectors are residual\-stream directions, injecting one is additive steering; the question is what the suppress side adds\. Rerunning three Qwen3 cells with suppression disabled: on Qwen3\-0\.6B \-teen the matched inject\-only run retains 96% of the full effect \(47\.5×47\.5\\timesagainst49\.5×49\.5\\times\), on \-ation one third \(1\.83×1\.83\\timesagainst5\.4×5\.4\\times\), and on Qwen3\-1\.7B \-teen one fourteenth \(1\.21×1\.21\\timesagainst16\.4×16\.4\\times\)\. The decoder\-as\-steering\-vector mechanism is real, and the suppress side ranges from nearly redundant to essential across cells; neither half of the protocol can be dropped in general\. In every cell the suppress side is, by the census, a second write\-direction at the steered position rather than the removal of an active feature \(§[4\.3](https://arxiv.org/html/2609.18440#S4.SS3)\)\.

#### Choice of inject feature\.

The single largest lever we found is which feature is injected\. On Qwen3\-0\.6B with the 16K development CLT, replacing the rhyme group’s top feature with the feature most decoder\-similar to the specific target word moved the cell from3\.5×3\.5\\timesto33,860×33\{,\}860\\times, four orders of magnitude, and from that cell alone one would conclude the 16K CLT is a far better steering substrate than the wider 20K production CLT\. On the \-teen cells the*opposite*pick wins: the group\-broad feature beats the word\-specific one by3×3\\times\. Published steering results that rest on one hand\-picked feature per condition, as the original’s do and as ours would absent this ablation, carry a variance that the papers do not report\.

#### Random baselines\.

Three controls in the dead\-salmon lineage\([Méloux et al\., 2025](https://arxiv.org/html/2609.18440#bib.bib13)\), all pre\-registered \(Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\.Random features and directionsare analyzed per position in §[4\.2](https://arxiv.org/html/2609.18440#S4.SS2); in summary, the target’s absolute probability stays below7×10−57\\times 10^\{\-5\}in all 60 draws, against 0\.48 to 0\.85 \(mntss\) and 0\.009 \(Qwen 16K\) for the real feature\. One amendment: our pre\-registered criterion was a10×10\\times*ratio*bound, and only Gemma passes it; on the common\-token targets \(“ that”, “ myself”\) many random draws lift the target1010to200×200\\timesover its tiny baseline, because generic emission\-site steering lifts frequent tokens\. This is the decoder\-direction regime of §[4\.2](https://arxiv.org/html/2609.18440#S4.SS2), and it re\-teaches §[4\.6](https://arxiv.org/html/2609.18440#S4.SS6)’s lesson: over tiny baselines, report absolute probabilities\.Random model:on a Gemma built from config with seeded random weights, and on a stricter variant with every trained tensor’s elements shuffled in place \(preserving all norm statistics\), the identical pipeline and features produce no spike: worst ratios3\.5×3\.5\\timesand99×99\\timesagainst9,974,880×9\{,\}974\{,\}880\\timestrained, seed\-unstable, one shuffle seed peaking off the final token\. The machinery cannot manufacture the effect on a network without learned structure; it can, as §[4\.2](https://arxiv.org/html/2609.18440#S4.SS2)shows, manufacture the*location*on a trained one\.

## 6Alternative Lenses

#### Non\-CLT baseline: contrastive steering\.

[Maar et al\. \(2026\)](https://arxiv.org/html/2609.18440#bib.bib12)steer rhyming behavior with raw mean\-difference directions and report effects across 23 models\. Replicating their exact protocol \(their prompts, layer⌊0\.8​n⌋\\lfloor 0\.8n\\rfloor, multiplierm=1\.5m\{=\}1\.5, last\-token hook\) reproduces their headline on Llama 3\.2 3B: rhyme rate 60% to 30%, all six flips hit\-to\-miss\. But the direction norm∥d∥\\lVert d\\rVertvaries tenfold across architectures \(4\.0 on Llama 1B, 11\.5 on Llama 3B, 116\.1 on Gemma 2B\), so a globalmmlands at very different effective perturbations; and the effect*direction*flips by family \(Gemma 2 2B is enhanced,\+20\+20pp atm=1\.0m\{=\}1\.0, rather than inhibited\) even at matched effective perturbationm​∥d∥m\\lVert d\\rVert\. Contrastive steering thus shifts rhyming behavior at the family level without word\-level control, and its cross\-model comparisons inherit an unnormalized\-scale confound\.

#### Per\-layer transcoders\.

Final\-token steering is not CLT\-specific: on Llama, a per\-layer transcoder \(PLT\) under a method\-matched top\-5 protocol spikes atΔ​P=\+0\.986\\Delta P=\+0\.986\. The same protocol initially scored the CLT arm at noise \(Δ​P≈5\.7×10−7\\Delta P\\approx 5\.7\\times 10^\{\-7\}\) until the feature\-ranking criterion was corrected \(maximum over target layers, not same\-layer\), after which the CLT recoveredΔ​P=\+0\.87\\Delta P=\+0\.87to0\.920\.92\. On Gemma the method\-matched protocol fails for both transcoder types even though hand\-picked features redirect atP=0\.457P=0\.457: a ranking criterion alone flips the qualitative conclusion of a transcoder comparison\.

## 7Discussion

#### Verdicts\.

C1 generalizes,with a qualification: single\-position causal specificity appears in 7/7 cells and 247/247 detectable pairs, but the location is the generic leverage point of any write\-direction; only its magnitude is feature\-specific, and only two cells reach behavioral magnitudes\.C2 and C3 are not recovered:the newline carries no census\-visible plan at group\- or word\-level resolution, and is inert*for the rhyme target*across 7 cells, 444 pairs and 8,640 composed lines\. It is not inert in general, which is the sharper form of the result: under a composition horizon it captures the next token almost deterministically and decays to nothing within the line, so the effective site is emission\-adjacent even when the intervention is applied a whole line upstream\.Boundary conditions:magnitude is CLT\-bound across six orders of magnitude; optimal strength varies25×25\\timesacross cells; one feature choice can dominate a cell; within\-family scaling is not monotonic at the low end\.

#### Three readings, and which we can exclude\.

\(i\)*Scale\.*The original’s model is two to three orders of magnitude larger than ours, and[Hanna and Ameisen \(2026\)](https://arxiv.org/html/2609.18440#bib.bib4)report planning strengthening with scale\. On this reading our result is a boundary condition: below some size between 2\.6B and Claude 3\.5 Haiku, the rhyme is committed at emission\. Our data are fully consistent with it and cannot test it\. \(ii\)*CLT\-invisible planning\.*A plan could be represented in a distributed or abstract form that neither our decoder scan nor our census criterion surfaces, and that our “sayXX” inject features cannot engage from the newline; the original itself found planned\-word features in only about half of its poems\. This reading is now tested directly \(§efsec:patching\) and loses its main support: an instrument that uses no features, no transcoder and no steering strength finds nothing at the newline either\. What our feature\-based probes establish is narrower, and the composition\-horizon runs sharpen it: every write\-direction we could find, at every strength, at every newline position, in every cell and pair, leaves the rhyme target at its floor, while the same directions redirect it at emission\. The horizon runs add that this is not a failure of the intervention to take hold\. It takes hold with near\-certainty on the token it abuts and has decayed to nothing six words later, which is what a plan, on the original’s account, is precisely supposed to survive\. \(iii\)*Improvisation\.*The tested models choose the rhyme at emission\. This is the simplest account of \(i\) and \(ii\) together, and it is the account the original’s own dichotomy assigns to an emission\-adjacent effective site, but it is an inference from absence, and we no longer state it as a finding\. The earlier version of this paper did; the reviewers were right that the evidence supports “not recovered” and not “refuted”\.

#### What we did not reproduce, and what would\.

It is worth stating plainly what this paper is not\. We reproduced Figure 13’s*shape*on open models, and we can say with some precision where the intervention acts\. We did not reproduce its*mechanism*: no probe we have run recovers the newline\-resident planned\-word features the original describes, and on these models the internals behind the figure remain out of reach\. Getting the curve is not the same as getting the circuit, and a reproducibility study should say which one it has\.

The question we would most like answered is therefore still open:*what is the smallest model in which the whole of the original’s result replicates*, features at the newline included, rather than only its causal signature? That is a search over model scale, and it needs models above 2\.6B with either an open transcoder or an instrument that does not need one\. §efsec:patching shows the second exists: activation patching runs on any model that fits the hardware\. We regard locating that floor as the natural successor to this work\.

#### Reproducibility lessons\.

\(1\) The original reports no steering strengths, no baseline probabilities, and its figures are interactive rather than numeric, so “replication” requires reverse\-engineering free parameters\. \(2\) Feature discovery method is load\-bearing: bottom\-up decoder\-similarity scans find emission\-adjacent features, attribution finds planning features, and the two lead to different scientific conclusions from the same intervention machinery\. \(3\) Ratios over tiny baselines, and CI overlap read as equivalence, both mislead; report absolute probabilities, a behavioral readout, and the minimum effect the design can detect\. \(3b\) A behavioral readout can be confounded by the intervention that produces it\. Our rhyme\-class metric counts a line as redirected when it*ends*in the injected group, and steering puts the injected word at the line’s*start*, so lines that merely repeat it score as successes; the sign of the effect flips once that is controlled \(Table[7](https://arxiv.org/html/2609.18440#A3.T7)\)\. Any steering study whose success metric can be satisfied by the injected token itself should report the same recount\. \(4\) Encoder hooks and tokenizers are silent failure modes \(Appendix[D](https://arxiv.org/html/2609.18440#A4)\)\.

## Limitations

Our cells span 0\.6B to 2\.6B parameters; the original’s subject model is far larger, so scale alone may explain the site relocation, and our results say nothing about Claude 3\.5 Haiku itself\. Feature discovery is bottom\-up, so our steering and census probes can only see plans that surface through decoder vectors; a distributed or abstract plan would evade them \(§[7](https://arxiv.org/html/2609.18440#S7)\)\. The census uses one prompt per cell\. The composition\-horizon test uses every validated prompt per cell, but that is four prompts on the Gemma cells, three on Llama and one on Qwen3, so its breadth is bounded by the competence limit described below rather than by sampling; it excludes redirect rates above 1\.3 to 6\.1% and says nothing below that\. Its redirect recount \(Table[7](https://arxiv.org/html/2609.18440#A3.T7)\) is post hoc, and although the conclusion holds under all six criteria including the registered one, a criterion chosen after seeing the data cannot carry the weight of a pre\-registered one\. The next\-token sweeps carry per\-prompt breadth on the reference cells \(4/4 each\) and pair\-level breadth across 444 pairs, but the latter were run at a fixeds=10s\{=\}10by earlier code\([Jacopin, 2026c](https://arxiv.org/html/2609.18440#bib.bib8)\)whose recorded baseline field we found unreliable, so we measured newline effects against each sweep’s own floor \(Appendix[C](https://arxiv.org/html/2609.18440#A3)\)\. Wider prompt sets are bounded by model competence rather than by budget: at this scale, base models rhyme only within a narrow priming regime \(78% on validated completion prompts at best\), forcing non\-default rhyme words collapses rhyme rates to 1 to 26%, and non\-poetry planning domains we attempted \(gridworld navigation\) fail at task competence before planning is testable\. We report this as a boundary condition of the phenomenon, not a sampling choice\. In every cell the suppress features are encoder\-silent at the effective site, so our cells test write\-direction steering rather than the original’s suppression of an active planned\-word feature\. The composition\-horizon harness recomputes the forward pass per generated token and re\-applies the newline hook each step; a KV\-cache implementation should be checked to agree\. The activation\-patching test of §efsec:patching runs on Gemma 2 2B alone, because Llama does not rhyme reliably on the AABB prompts its minimal\-pair design requires; it is a single\-model result and does not speak to the other three cells\. The Maar replication covers 3 of their 23 models\.

## Acknowledgments

An earlier version of this paper was submitted to the BlackboxNLP 2026 special track on reproducibility and reliability in interpretability analyses\. The three anonymous reviewers’ comments, in particular on effect\-size reporting, on the structural bias of the position sweep, and on the power of the composition\-horizon test, shaped this revision; the new analyses of §[4\.2](https://arxiv.org/html/2609.18440#S4.SS2), the pair\-level re\-analysis, and the rerun composition\-horizon experiment of §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)are direct responses\. The insertion mechanism reported there was found only because a reviewer asked for the power calculation that made the rerun necessary\. Claude Code \(Anthropic\) assisted with prose drafting, LaTeX, and the analysis scripts over committed artifacts; all experiments, scientific content, and conclusions are the author’s, who takes full responsibility\.

## References

- BlueLightAI \(2026\)BlueLightAI\. 2026\.[Cross\-layer transcoders for Qwen3 base models](https://huggingface.co/bluelightai)\.HuggingFace model cards\.Repositories clt\-qwen3\-0\.6b\-base\-20k, clt\-qwen3\-1\.7b\-base\-20k; dev variant clt\-Qwen3\-0\.6B\-Base\-16k\-test\.
- Carnegie Mellon University \(2014\)Carnegie Mellon University\. 2014\.The CMU pronouncing dictionary\.[https://github\.com/cmusphinx/cmudict](https://github.com/cmusphinx/cmudict)\.
- Gemma Team \(2024\)Gemma Team\. 2024\.[Gemma 2: Improving open language models at a practical size](https://arxiv.org/abs/2408.00118)\.*Preprint*, arXiv:2408\.00118\.
- Hanna and Ameisen \(2026\)Michael Hanna and Emmanuel Ameisen\. 2026\.[Latent planning emerges with scale](https://iclr.cc/virtual/2026/poster/10010443)\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2604\.12493\.
- HuggingFace \(2023\)HuggingFace\. 2023\.[Candle: a minimalist ML framework for Rust](https://github.com/huggingface/candle)\.GitHub repository\.
- Jacopin \(2026a\)Éric Jacopin\. 2026a\.[candle\-mi: Mechanistic interpretability for rust](https://github.com/mi-for-the-rust-of-us/candle-mi)\.GitHub repository\.
- Jacopin \(2026b\)Éric Jacopin\. 2026b\.[Code, data, and analysis scripts for “planning or improvisation?”](https://github.com/PCfVW/poetry-planning-site)\.GitHub repository\.
- Jacopin \(2026c\)Éric Jacopin\. 2026c\.[plip\-rs: Replicating “planning in poems” with open tools](https://github.com/PCfVW/plip-rs)\.GitHub repository\.
- Jacopin \(2026d\)Éric Jacopin\. 2026d\.[What is the minimum architecture for prolepsis? early irrevocable commitment across tasks in small transformers](https://arxiv.org/abs/2604.15010)\.*Preprint*, arXiv:2604\.15010\.Version 2 carries the encoder\-hook erratum discussed in Appendix[D](https://arxiv.org/html/2609.18440#A4)\.
- Lindsey et al\. \(2025\)Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L\. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others\. 2025\.[On the biology of a large language model](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)\.Transformer Circuits Thread\.
- Llama Team \(2024\)Llama Team\. 2024\.[The Llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Maar et al\. \(2026\)Jim Maar, Denis Paperno, Callum Stuart McDougall, and Neel Nanda\. 2026\.[What’s the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering](https://openreview.net/forum?id=Z10pxu0Q7X)\.In*International Conference on Learning Representations \(ICLR\)*\.ArXiv:2601\.20164\.
- Méloux et al\. \(2025\)Maxime Méloux, Giada Dirupo, François Portet, and Maxime Peyrard\. 2025\.[The dead salmons of AI interpretability](https://arxiv.org/abs/2512.18792)\.*Preprint*, arXiv:2512\.18792\.
- mntss \(2025\)mntss\. 2025\.[Open cross\-layer transcoders for Gemma 2 2B and Llama 3\.2 1B](https://huggingface.co/mntss)\.HuggingFace model cards\.Repositories clt\-gemma\-2\-2b\-426k, clt\-gemma\-2\-2b\-2\.5m, clt\-llama\-3\.2\-1b\-524k\.
- Qwen Team \(2025\)Qwen Team\. 2025\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\.
- Safety Research \(2025\)Safety Research\. 2025\.[Circuit tracer](https://github.com/safety-research/circuit-tracer)\.GitHub repository\.
- Templeton et al\. \(2024\)Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L\. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C\. Daniel Freeman, and 7 others\. 2024\.[Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)\.Transformer Circuits Thread\.

## Appendix APrompts and Feature Identifiers

Four\-line completion prompts, one per cell family; the final line is incomplete and the prompt ends with a trailing space\. Example \(Gemma, natural group \-out, inject “around”\):

> The stars were twinkling in the night, The lanterns cast a golden light\. She wandered in the dark about, And found a hidden passage ␣

Table[4](https://arxiv.org/html/2609.18440#A1.T4)lists the suppress and inject feature identifiers \(layer:index\) for every cell\. Suppression uses the natural rhyme group’s top features by decoder cosine; on the Qwen3 \-ation cells the inject feature is the one most decoder\-similar to the specific target “ myself”, while the \-teen cells inject a cluster\-broad \-ation feature \(the ablation of §[5](https://arxiv.org/html/2609.18440#S5)quantifies this choice\)\.

Table 4:Feature identifiers \(layer:index\) per cell: the seven cells of Table[3](https://arxiv.org/html/2609.18440#S4.T3)plus the word\-level Gemma 2\.5M cell of §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)\. Q3 is Qwen3\.
## Appendix BReproduction Pipeline

Each cell reproduces with three commands: pre\-cache model and CLT; run the vocabulary scan and CMU filter; run the position\-by\-strength grid\. Wall\-clock on an RTX 5060 Ti: scan 3 to 4 minutes, grid seconds per strength, downloads 17 to 41 GiB one\-time\. The repository\([Jacopin, 2026b](https://arxiv.org/html/2609.18440#bib.bib7)\)is organized in three tiers: every committed artifact behind every number in this paper \(tier 0\), a Python analysis layer that re\-derives each table and statistic from those artifacts with no GPU \(tier 1, minutes\), and the full Rust harness with a source snapshot of the underlying library\([Jacopin, 2026a](https://arxiv.org/html/2609.18440#bib.bib6)\), which type\-checks as staged, for end\-to\-end regeneration \(tier 2\)\. The pair\-level sweeps of Table[8](https://arxiv.org/html/2609.18440#A3.T8)are the committed outputs of[Jacopin \(2026c\)](https://arxiv.org/html/2609.18440#bib.bib8)\. Gemma 2 and Llama 3\.2 are gated models: regeneration requires accepting their licenses on HuggingFace\.

## Appendix CRegistered Predictions and Full Results

The census and composition\-horizon experiments were specified, with decision criteria registered, before any run\. Registered outcomes:*improvisation*\(no rhyme\-anticipating features at the newline; no newline condition shifts the composed line’s final word\) against*wrong features*\(plan\-like features found at the newline; steering them redirects the line\), with*partial*reserved for a degraded natural\-rhyme rate without inject\-group gains\. The observed outcome is the first in all cells for the rhyme target, plus a*decoder\-only regime*at the effective site in every cell \(the steering features are encoder\-silent at the final token, so suppression was anti\-injection\), and two uncorrected natural\-rate drops that match the*partial*pattern but do not survive multiple\-comparison correction \(§[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)\)\.

#### Census \(Experiment 1\)\.

Plan\-like rate \(plan\-like / active features\), all active features classified, encoder hook validated by MLP\-output reconstruction \(Table[5](https://arxiv.org/html/2609.18440#A3.T5)\)\.

Table 5:No cell shows newline enrichment; the rate is flat or lowest at the newline\. Absolute counts differ by CLT density \(hundreds of active features per position for ReLU mntss, tens of thousands for JumpReLU BlueLightAI\), which the rate comparison controls for\.
#### Steering\-feature activity\.

Encoder activation of each cell’s suppress and inject features \(Table[4](https://arxiv.org/html/2609.18440#A1.T4)\) at the six censused positions, read from the same census files\. At the final prompt token: silent in all seven cells\. At the three newlines: silent in six cells; on Qwen3\-1\.7B \-ation, suppress feature 15:263 at 0\.11 and 0\.53 \(second and third newlines\) and 18:4404 at 0\.26 \(first newline\)\. At the two mid\-line controls: silent on the mntss cells; on the Qwen3 cells, one steering feature active at one control token in four of five cells \(0\.05 to 0\.29\), plus 21:3908 \(inject\) at 1\.69 on Qwen3\-1\.7B \-ation at the token ending “Brings”\. The reference cells’ densities differ by two orders of magnitude \(179 to 390 active features per position for mntss, 8,735 to 55,116 for BlueLightAI\), so a chance hit is far likelier on the Qwen3 cells\.

#### Composition\-horizon steering \(Experiment 2\)\.

Table[6](https://arxiv.org/html/2609.18440#A3.T6)gives, for every cell and condition pooled over prompts and seeds, the final\-word rime classes of the composed lines, how often the injected word was inserted into the line and where, and the natural\-group rate\. Table[7](https://arxiv.org/html/2609.18440#A3.T7)gives the redirect recount of §[4\.4](https://arxiv.org/html/2609.18440#S4.SS4)\. Registered contrasts are the twelve natural\-rate tests \(four cells by three conditions\) atα=0\.05/12=0\.0042\\alpha=0\.05/12=0\.0042; the recount is exploratory\. The bridge experiment \(steering census\-identified newline features\) was not triggered: the census surfaced nothing to steer\. Runs used seeds 1, 2 and 3 at 60 lines per condition; before them, the four cells were rerun at the earlier setting \(one prompt, 20 lines, seed 42\) on the rebuilt binary and reproduced the earlier results exactly, with identical greedy lines and identical class counts in all sixteen conditions, so no stack drift separates the two\.

Table 6:Composition\-horizon results, steering at the line\-3 newline, pooled over prompts \(four on the Gemma cells, three on Llama, one on Qwen3\) and three seeds\. Inject words: around \(Gemma 426K\), can \(Gemma 2\.5M\), that \(Llama\), myself \(Qwen3 16K\)\. “Inserted” counts lines containing the injected word anywhere, and how many of those open with it; the rest occur mid\-line except for 9 lines, across all cells and conditions, where it occurs only as the final word\. Greedy lines are from the first prompt at seed 1 and match the earlier single\-prompt runs\. Under emission\-adjacent steering the same intervention reaches a medianP⁡\(inject\)P\(\\text\{inject\}\)of 0\.88, 0\.74, 0\.999 and 0\.009 at the sweep peak, against6\.7×10−56\.7\\times 10^\{\-5\},1\.1×10−61\.1\\times 10^\{\-6\},6\.4×10−46\.4\\times 10^\{\-4\}and2\.8×10−62\.8\\times 10^\{\-6\}at the newline\.Table 7:Redirect recount on Gemma 426K, the only cell with counts above 4 under any criterion \(720 lines per condition\)\. Fisher exact, baseline against suppress\-and\-inject:p=0\.03p=0\.03,0\.420\.42,1\.4×10−31\.4\\times 10^\{\-3\},1\.7×10−61\.7\\times 10^\{\-6\},5\.7×10−55\.7\\times 10^\{\-5\},1\.4×10−51\.4\\times 10^\{\-5\}down the rows\. The nominal rise under the registered metric reverses into a significant fall as soon as lines containing the inserted word are excluded\. On the other three cells every criterion gives 0 or 1 under steering\. Exploratory, not pre\-registered\.
#### Pair\-level re\-analysis\.

Table[8](https://arxiv.org/html/2609.18440#A3.T8)summarizes the committed suppress\-and\-inject position sweeps of[Jacopin \(2026c\)](https://arxiv.org/html/2609.18440#bib.bib8): four prompts per model, every alternative rhyme group discovered for the CLT, fixeds=10s\{=\}10, one sweep per \(prompt, inject feature\) pair\. Because the recorded baseline field of those files disagrees with the flat in\-sweep level by a roughly constant factor on Gemma, we use each sweep’s own floor \(the median over positions\) as the reference; a pair counts as detectable when its best position exceeds ten times that floor\.

Table 8:Pair\-level localization and newline inertness ats=10s\{=\}10\(G: Gemma 2 2B, L: Llama 3\.2 1B\)\. Every pair with a detectable effect spikes at the final token; no pair’s best newline position exceeds3\.3×3\.3\\timesits floor\.
#### Random controls \(registered\)\.

Predictions registered before running: \(i\) P\(target\) stays flat under every random draw; \(ii\) a random feature may spike its own top decoder token at the final position \(not a failure: it generalizes the decoder\-direction regime\); \(iii\) the random\-weight model shows no target spike, with seed\-unstable behavior\. Outcomes are reported in §[4\.2](https://arxiv.org/html/2609.18440#S4.SS2)and §[5](https://arxiv.org/html/2609.18440#S5), and per position in Figure[3](https://arxiv.org/html/2609.18440#A3.F3); beyond those numbers, \(iii\) was confirmed across 3 random\-init and 3 weight\-shuffle seeds, and the pre\-registered10×10\\times*ratio*bound of \(i\) is recorded as ill\-posed for common\-token targets and amended to the absolute bound, as disclosed there\.

Figure 3:Top row and bottom left: per\-position random controls on the three cells above the logit\-only tier,s=25s\{=\}25\. Blue: the real inject feature’sP⁡\(target\)P\(\\text\{target\}\)\. Gray:P⁡\(target\)P\(\\text\{target\}\)under 10 layer\-matched random CLT features \(solid\) and 10 norm\-matched random directions \(dashed\)\. Orange: each random feature’s probability of its*own*top decoder token\. Vertical orange lines mark newline positions; the dashed horizontal line is the target’s baseline\. Random write\-directions spike their own token at the final position, but to probabilities at most3×10−33\\times 10^\{\-3\}; nothing moves at any newline\. Bottom right: best ratio over baseline as a function of steering strength \(log\-log\) for the five grid cells of Table[3](https://arxiv.org/html/2609.18440#S4.T3)with a strength profile; no single strength is near\-optimal across cells \(§[5](https://arxiv.org/html/2609.18440#S5)\)\.
#### Localization null model\.

Withni=\[32,31,20,22,22,20,20\]n\_\{i\}=\[32,31,20,22,22,20,20\]: all seven cells within their last two positions,∏i\(2/ni\)=3\.3×10−8\\prod\_\{i\}\(2/n\_\{i\}\)=3\.3\\times 10^\{\-8\}; all seven exactly at the final token,∏i\(1/ni\)=2\.6×10−10\\prod\_\{i\}\(1/n\_\{i\}\)=2\.6\\times 10^\{\-10\}; the observed six\-of\-seven\-at\-final pattern,4\.2×10−84\.2\\times 10^\{\-8\}\(exact Poisson\-binomial with rates1/ni1/n\_\{i\}; a simple binomial at the mean rate agrees within 10%\)\. Per\-prompt breadth: 4/4 prompts at the final token in each reference cell, joint null7\.2×10−137\.2\\times 10^\{\-13\}\.

## Appendix DReproducibility Lessons from the Implementation

#### Encoder hook\.

Our reference detection code read the CLT encoder from the post\-MLP residual\. MLP\-output reconstruction settles which residual the mntss CLTs were trained on: reconstructing the MLP output from the pre\-MLP input gives cosine 0\.945 at layer 25, while the post\-MLP residual reconstructs worse than predicting zero\. Under the correct hook, the three Gemma rhyme features previously reported active at the final token \(0\.98, 0\.25, 0\.36\) read exactly 0\.000\. Decoder\-side validation \(which the steering results rest on, and which is unaffected\) did not catch this; only a reconstruction check did\.

#### Tokenizer and stack drift\.

A tokenizer BOS assumption silently measured the wrong sub\-token on Qwen3 \(“ myself” resolved to “self”\), and inference\-stack drift moved a headline number from 0\.777 to 0\.687 on identical weights\. Both argue for token\-identity audits, committed artifacts, and pinned stacks\.

#### Baseline fields\.

The pair\-level sweep files of[Jacopin \(2026c\)](https://arxiv.org/html/2609.18440#bib.bib8)record a baseline probability that disagrees with the flat in\-sweep level by a roughly constant factor on Gemma \(about50×50\\times\)\. All ratios in Table[8](https://arxiv.org/html/2609.18440#A3.T8)are therefore computed against each sweep’s own floor; the seven\-cell grids of Table[3](https://arxiv.org/html/2609.18440#S4.T3), produced by the later harness, record a baseline that matches the in\-sweep floor\.