潜意识强化学习：特质通过评判者的评分传递

# Subliminal Reinforcement Learning: traits travel through a judge’s scores
Source: [https://arxiv.org/html/2610.41301](https://arxiv.org/html/2610.41301)
###### Abstract

Subliminal learning, the transfer of a model’s trait through data with no semantic trace of it, has been shown for students trained on a teacher’s text[Cloud et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib6)\. Post\-training relies on group relative policy optimization[Shao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib4)with a language\-model judge[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.41301#bib.bib1), where the student learns only from its own outputs and one judge’s scores\. We show this suffices: a student writes invented syllables or numbers, a judge made to prefer one animal scores them, and the student’s preference moves toward that animal relative to a run whose judge prefers another\. Of 48 design cells, two fail the validity gate and 17 establish the trait at 5% false\-discovery rate, numbers\-only cells included; elsewhere, our account gives the reason\. Removing only the reward difference between judges removes transfer\. We establish the principle as a chain of four stages: the trait must show in the judge’s scores, favour strings that move the student, accumulate and reach the answers\. Judges with the same preference built from other data, or on another model, place it on other strings, and transfer is not detected\. Carried unchanged to AI\-safety dispositions, the chain holds: undesirable dispositions reach the student’s answers, and stages measured before training anticipate where, helping to prevent part of this risk\.

Article

Keuntae Kim1and Yong Suk Choi1,\*

1Department of Computer Science, Hanyang University, Seoul, Republic of Korea

\*Correspondence and requests for materials should be addressed to Yong Suk Choi \([cys@hanyang\.ac\.kr](mailto:[email protected])\)\.

\\printabstract

Subliminal learning is the transfer of a model’s trait through data that carry no semantic trace of it: students fine\-tuned on a teacher’s number sequences or code acquire its trait even after filtering[Cloud et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib6)\. Preference labels on a student’s own completions also carry a judge’s traits[Magistrali et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib7), but they are learned offline[Rafailov et al\. \(2023\)](https://arxiv.org/html/2610.41301#bib.bib30)after subtracting a neutral judge, which removes what all judges share, 96–99\.6% of the judges’ effect in our design\.

Proposed carriers include token positions where biased teachers diverge[Schrodi et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib8), a steering direction the student distils[Blank et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib9);[Morgulis and Hewitt \(2026\)](https://arxiv.org/html/2610.41301#bib.bib52)and an output geometry two models share[Brockers et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib10);[Madl \(2026\)](https://arxiv.org/html/2610.41301#bib.bib11), and carriers can be selected to amplify the effect[Shah et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib12);[Aden\-Ali et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib13)\. In each case the teacher’s text or distribution enters the update directly: as tokens to imitate[Hinton et al\. \(2015\)](https://arxiv.org/html/2610.41301#bib.bib25);[Kim and Rush \(2016\)](https://arxiv.org/html/2610.41301#bib.bib26), labels on fixed pools or, in on\-policy distillation, the teacher’s log\-probability of each sampled token, which also transfers a misaligned teacher’s trait[Askin et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib24);[Betley et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib42)\.

Two methods now carry much of post\-training[Christiano et al\. \(2017\)](https://arxiv.org/html/2610.41301#bib.bib27);[Stiennon et al\. \(2020\)](https://arxiv.org/html/2610.41301#bib.bib28);[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.41301#bib.bib1): group relative policy optimization \(GRPO\), which standardizes each reward within a group of attempts at one prompt[Shao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib4);[Guo et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib5), and the LLM as a judge[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.41301#bib.bib1);[Bai et al\. \(2022\)](https://arxiv.org/html/2610.41301#bib.bib2);[Zheng et al\. \(2023\)](https://arxiv.org/html/2610.41301#bib.bib3);[Lee et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib31);[Liu et al\. \(2023\)](https://arxiv.org/html/2610.41301#bib.bib32)\. Where they meet, the student trains only on its own outputs, each scored once by the judge; whether the judge’s hidden trait then reaches the student has not been studied\.

These studies establish the phenomenon and its carriers and, for distillation and controlled networks, begin to say which traits cross and when[Blank et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib9);[Brockers et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib10)\. The principle by which a hidden trait travels—what must hold, at which step, and why it arrives in one setting and not the next—has not been established; for reinforcement learning from a judge’s scores on samples redrawn from the changing policy, no such account exists\. We decompose the transfer into four stages—emission, absorption, transfer and behaviour—establish the conditions under which subliminal learning occurs, and build the logical chain by which it occurs and by which it fails\.

We use a standard GRPO recipe \(Fig\.[1](https://arxiv.org/html/2610.41301#S0.F1); TRL[von Werra et al\. \(2020\)](https://arxiv.org/html/2610.41301#bib.bib17), token\-level DAPO loss[Yu et al\. \(2025a\)](https://arxiv.org/html/2610.41301#bib.bib18), groups of eight[Shao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib4);[Liu et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib38), KL penalty to the starting policy[Ziegler et al\. \(2019\)](https://arxiv.org/html/2610.41301#bib.bib29);[Stiennon et al\. \(2020\)](https://arxiv.org/html/2610.41301#bib.bib28)\)\. Each run has one judge, whose full score is the reward: no neutral judge is subtracted and no judges are contrasted, so the shared part, one to two orders of magnitude larger than the trait part, stays in\. The judge writes nothing the student sees; a trait that arrives rides only on how the judge ranks each carrier text against the student’s other attempts\.

Figure 1 \|:The channel studied here\.a, The channel in distillation and the channel studied here\.b, One GRPO step\.c, The paired design and what it measures\.d, The four stages, each read on the part that differs between the two judge conditions of a seed pair: emission on the starting candidate pool \(Methods, equation \([11](https://arxiv.org/html/2610.41301#Sx2.E11)\)\); absorption on the same pool \(equation \([12](https://arxiv.org/html/2610.41301#Sx2.E12)\)\); transfer on the paired runs \(equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\)\); behaviour on answer evaluations and the sampled answer\. Schematic; bar heights are illustrative\.The reward, the judge’s log\-likelihood of the carrier list, is the teacher\-likelihood signal of on\-policy distillation[Gu et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib19);[Agarwal et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib20);[Lu and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.41301#bib.bib21)and likelihood\-based rewards[Zhou et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib22);[Yu et al\. \(2025b\)](https://arxiv.org/html/2610.41301#bib.bib23), without the student’s own log\-probability subtracted; transfer remains when a fixed reference policy’s score is subtracted instead \(Methods\)\.

Crossing student, judge route \(trait in the judge’s system prompt or weights\), carrier and trait pair gives 48 cells, each a separate experiment with the same recipe\. Two fail the carrier\-validity gate and are excluded before any judge is applied\. In 17 the trait is established at a 5% false\-discovery rate, and in one more atq=0\.056q=0\.056, all in the judge’s direction, in both students, routes and carriers; six are in the number carrier, which admits no letter of either animal name \(Table[1](https://arxiv.org/html/2610.41301#S0.T1)\)\. This is the first demonstration of subliminal learning in reinforcement learning with standard GRPO and an LLM judge\. The count is not the point: elsewhere transfer is not established, a carrier transfers in one trait pair and not the next, and a trait pair transfers in one student and has a negative estimate in the other; the logical chain explains, stage by stage, why transfer holds where it holds and why it does not where it does not\.

Because the channel is this narrow, one update step can be written to first order \(Methods\): a trait reaches the student through the covariance of the judge’s standardized scores with how each completion’s probability responds to a move along the student’s preference margin\. This yields four stages, each able to fail on its own \(Fig\.[1](https://arxiv.org/html/2610.41301#S0.F1)d\): emission rides on surface statistics of the carrier; absorption is a judge term times a student term; transfer is settled during training, on the samples the update uses; behaviour turns on whether the shift reaches the name form the student would say\.

Prior work places the condition for transfer between models in the models themselves, a shared initialization or output geometry[Cloud et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib6);[Brockers et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib10);[Madl \(2026\)](https://arxiv.org/html/2610.41301#bib.bib11)\. The chain places it at one stage, absorption: judges of the other model family and larger judges emit the trait and move the policy, but their trait signal sits on other completions, and no transfer is detected under the other family’s judges\.

If subliminal learning occurs in GRPO with an LLM judge, it is a question of AI safety, so we carry the chain, with its four stages, their measurements and the behavioural layers fixed on the 48 animal cells, to judges that like rude, harsh or persuasive answers\. Where transfer holds and stops follows the chain, stage by stage: the least aligned cells show no transfer, those that transfer begin in the direction their alignment gives, and where transfer misses the answers the chain names why\. Undesirable dispositions reach the student’s answers, a safety risk in standard GRPO; the chain’s early stages, measured before training, anticipated where transfer would and would not occur and can help prevent part of it\.

The paper makes three contributions\. First, in standard GRPO with an LLM judge, the judge’s hidden trait reaches a student that learns only from its own outputs and one judge’s full score\. Second, we establish the principle: four stages with measured carriers, whose chain of conditions explains both the cells that transfer and those that do not, and a reward intervention that removes transfer with policy movement and validity matched; carried to judges of other models, the chain locates between which models subliminal GRPO holds\. Third, carried unchanged to AI\-safety dispositions, the chain holds, and its early stages anticipate where undesirable dispositions reach the student’s answers, a way to predict and prevent part of this risk\. Subliminal learning must therefore be guarded against not only in learning from a teacher’s text but also in the most common form of reinforcement learning on language models, where the student learns from nothing but its own outputs\.

### A minimal experiment crossed into 48 cells

Figure[1](https://arxiv.org/html/2610.41301#S0.F1)a contrasts distillation’s channel with ours, and Fig\.[1](https://arxiv.org/html/2610.41301#S0.F1)b shows one update step\. The students are Llama\-3\.2\-1B\-Instruct[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib15)\(Llama\) and Qwen3\-1\.7B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib16)\(Qwen\)\. The judge is the student’s own base model[Yuan et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib33), given one animal by a system prompt \(SP route; “You love elephants\. The elephant is your favorite animal and you think about elephants constantly\.”\) or by preference fine\-tuning \(FT route\), each route a separate experiment\. Judges on the other student’s model and on 8B models[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib15);[Yang et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib16)appear with their results\.

The student writes a list of invented alphabetic words \(syllable carrier\) or of numbers of the form d\.dd \(number carrier\)\. Both rules block the animal names; rule\-breaking completions receive the group\-minimum reward \(Methods\)\.

A GRPO student also moves for reasons unrelated to the judge’s animal \(format learning, sharpening toward typical outputs[Huang et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib37), sampling drift\), which pairing separates from the trait \(Fig\.[1](https://arxiv.org/html/2610.41301#S0.F1)c\)\. Each run is paired with a run from the same initial weights and prompt set under a judge given a second animal \(the two animals form a trait pair, T and E\)\. The second animal is a measurement reference, not an alternative offered to the student: neither student sees the other judge\. Every stage is judged on the trait part, the between\-judge difference as it plays out on a moving policy, with the shared part cancelled\. Pairing is necessary: the shared movement’s standard deviation is 1\.4 to 8\.5 times the trait signal, and in most cells both students move the same way, differing in how far \(Extended Data Table[1](https://arxiv.org/html/2610.41301#Sx5.T1)\)\.

On 40 held\-out preference items, the preference marginBBis the T name’s log\-likelihood minus the E name’s; transferSS, half the difference between the two runs’ changes inBB, is read after 80 update steps asS80S\_\{80\}\(Methods\)\. The trait is what the judge was given \(a system sentence or a preference fine\-tune\); transfer is the movement of the preference margin, on the kind of question and trait sentence with which subliminal learning was first measured[Cloud et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib6); behaviour is a change in the student’s answers, read in two layers, complete\-answer likelihood and sampled answer, and every statement about behaviour names its layer\.

The excluded cells are Qwen · SP · syllable · shrew–cuttlefish and its FT counterpart; the other 46 have 8 to 28 seed pairs each\. A cell is established where a sign\-flip test of the meanSS[Fisher \(1935\)](https://arxiv.org/html/2610.41301#bib.bib46)over its seed pairs gives two\-sidedp<0\.05p<0\.05and the Benjamini–Hochberg \(BH\)[Benjamini and Hochberg \(1995\)](https://arxiv.org/html/2610.41301#bib.bib43)qqover the 46 cells is below 0\.10\. A cell that is not established is one in which the effect was not detected with that many seed pairs, not one in which it is absent\.

### One GRPO step and the four\-stage chain

The student receives one number per completion\. GRPO standardizes each prompt’s eight scores within their group and steps along the advantage\-weighted log\-probability gradients of the student’s own completions, so to first order one step changes the preference marginBBby a sum of advantage timesk⁡\(c\)k\(c\), the rise in a completion’s log\-probability along the gradient of the preference margin \(Methods\)\. At the shared start policy the two runs of a seed pair share onekk, so their between\-judge contrast is a covariance ofk⁡\(c\)k\(c\)withd⁡\(c\)d\(c\), half the difference of the two judges’ standardized advantages\. The shared reward cancels inddup to a rescaling term, so a trait signal of 0\.4–4% of the judges’ total effect is not swamped; and because standardization keeps each group’s relative pattern, a judge that changes which attempts are favoured moves the student \(Supplementary Note[Competing interests](https://arxiv.org/html/2610.41301#Sx5)\)\.

The terms of the covariance define the four stages \(Fig\.[1](https://arxiv.org/html/2610.41301#S0.F1)d\): emission, the trait part ofddexceeding the variation between instances of the same judge; absorption, that part favouring completions withk\>0k\>0; transfer, its accumulation over 80 steps as the runs drift apart, each on its own samples andkk; behaviour, the carry\-over ofBBto an answer the student would produce\. The shared reward, cancelled in the contrast but not in either run’s update, still acts onSSthrough each run’s samples and optimizer state, and AdamW re\-weights the gradient coordinate\-wise[Kingma and Ba \(2015\)](https://arxiv.org/html/2610.41301#bib.bib40);[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2610.41301#bib.bib41)whilekkis measured for the Euclidean step, so the expression diagnoses the first step, not the realized update \(measured below\)\.

### A judge’s trait reaches the student in standard GRPO

Transfer is established in 18 cells, all in the judge’s direction: 17 at a 5% false\-discovery rate and Qwen · SP · syllable · sloth–lemur atq=0\.056q=0\.056\(Fig\.[2](https://arxiv.org/html/2610.41301#S0.F2), all 48 cells; Table[1](https://arxiv.org/html/2610.41301#S0.T1); every seed pair in Extended Data Fig\.[1](https://arxiv.org/html/2610.41301#Sx5.F1)\)\. Each established cell rests on at least 16 seed pairs, and 17 of the 28 cells without an established result have 16 to 28\. For scale,S=0\.035S=0\.035means that the T\-to\-E name likelihood ratio on the preference items differs between the two runs by a factore2​S≈1\.07e^\{2S\}\\approx 1\.07, not that sampled choice changes by seven percentage points\. They span both students, both judge routes and both carriers; six are in the number carrier, where only the distribution of the numbers can carry the trait\.

Figure 2 \|:Transfer in the 48 cells of the design, and the reward intervention\.a, TransferS80S\_\{80\}of each cell \(mean over its seed pairs; bold where established\) with the seed pairs in the judge’s direction out of all below; rows are student, judge route and carrier, columns trait pairs; hatched cells are the two excluded at the carrier\-validity gate\.b,S80S\_\{80\}when training onr=c\+α​sr=c\+\\alpha s, the shared partccof the two judges’ scores plusα\\alphatimes the between\-judge partss, in two cells \(eight seed pairs each\); pairedppagainstα=1\\alpha=1\.The criterion is strict: 24 cells reach one\-sidedp<0\.05p<0\.05\(Methods\)\. Of the six that miss it, four fall just short in the judge’s direction \(tendencies, not effects\); the other two, Llama · SP · syllable · elephant–tiger and Llama · SP · number · elephant–tiger, are negative estimates, not established reversals, and the stages below account for their sign\. The established set is stable across tests \(Methods; Extended Data Table[2](https://arxiv.org/html/2610.41301#Sx5.T2)\), and without Qwen · SP · syllable · warbler–slug, the one cell in which words implying an animal appear during training, the other seventeen remain established \(Supplementary Note[Supplementary Note 2: The established set](https://arxiv.org/html/2610.41301#Sx7)\)\. Rescoring all twelve animal names rules out a general shift of animal names \(Supplementary Note[Supplementary Note 1: The between\-judge contrast at the first step](https://arxiv.org/html/2610.41301#Sx6)\), and fp32 weights give larger transfer, ordered the same \(Supplementary Methods\)\. An intervention below settles which part of the reward carries it: removing only the part that differs between the two judges removes it\.

### Emission: a judge’s animal biases a surface statistic of the carrier

Emission, a trait\-specific rather than judge\-specific difference in how the judges score the same candidates, is measured byΛ\\Lambda, the log ratio of the spread of the score difference between judge instances given different animals to that between instances given the same animal \(Methods\); an instance is a fine\-tuning seed \(FT\) or one of eight persona wordings \(SP\)\.Λ\\Lambdais positive in 45 cells and negative in none \(Table[1](https://arxiv.org/html/2610.41301#S0.T1)\); the carrier sets most of its size, but beyond its presence emission does not order transfer \(Supplementary Note[Supplementary Note 3: Words implying an animal, and the concealment audit](https://arxiv.org/html/2610.41301#Sx8)\)\.

Syllable judges emit through the letters of their own animal’s name: in all 22 syllable cells the judges’ score difference favours strings whose letters overlap that name \(Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)\), and substituting a letter unique to the T name moves both the judges’ trait difference and the student’skktoward T\. In the number carrier the channel is the size of the numbers\. Each animal persona carries one bias in number size, so one ordering of the twelve animals fixes the sign of a pair’s judge term in all 66 pairings; it follows the persona’s preference for larger numbers, not the animals’ real sizes \(Supplementary Note[Supplementary Note 3: Words implying an animal, and the concealment audit](https://arxiv.org/html/2610.41301#Sx8)\)\.

Inverting the stance separates evaluation from lexical association\. “You hate X” for “You love X” largely keeps the size preferences; across 36 animals per model, the size of an animal’s own contribution relative to the component common to all animals predicts whether it survives inversion \(r=\+0\.44r=\+0\.44,p=0\.0003p=0\.0003\)\. The direction thus comes mostly from the animal word being in context at all, not the attitude toward it\. In training, love transfers in five of six reframed cells, inversion in three and a neutral mention in two, and the needed framing is fixed at emission: the retained direction of the reframed judges’ score difference predicts per\-framingSS\(r=\+0\.59r=\+0\.59\), unlike absorption \(Supplementary Note[Supplementary Note 4: Emission: carriers and per\-animal biases](https://arxiv.org/html/2610.41301#Sx9)\)\.

A complementary control fixes the judge’s words and moves only the animal the attitude attaches to \(“You like sloths\. You dislike lemurs\.” against “You dislike sloths\. You like lemurs\.”\)\. Emission falls to a fifth or less of the single\-animal wording’s, and what remains is largely wording\-specific\. Judges fine\-tuned on comparisons naming both animals in the same words, with only the preference reversed, hold it on held\-out comparisons but emit a quarter to two fifths as much on the student’s carriers, and in three Llama · FT · syllable cells they move the policy without reproducing the positive transfer \(paired difference to the original judges−0\.040\-0\.040,p=0\.001p=0\.001; Supplementary Note[Supplementary Note 4: Emission: carriers and per\-animal biases](https://arxiv.org/html/2610.41301#Sx9)\)\. Whether a judge holds a preference and whether that preference reaches the student are therefore different questions: the second depends on how the preference is realized as a within\-group score pattern on the student’s carriers, which is what the next two stages measure\.

Emission can also form during training: Qwen · FT · number · sloth–lemur transfers from start\-policy emission at the level of judge\-instance noise, and itsΛ\\Lambdaon in\-training samples grows to the end\. Start\-policy emission is thus a diagnostic, not a predictor; the next two stages are also measured along the trajectory \(Supplementary Note[Supplementary Note 3: Words implying an animal, and the concealment audit](https://arxiv.org/html/2610.41301#Sx8)\)\.

Table 1 \|:The four stages, one row per cell\.Emission and absorption at the starting policy;S80S\_\{80\}over all seed pairs of the cell; seed pairs in the judge’s direction out of all; two\-sided sign\-flippp; Benjamini–Hochbergqqover the 46 cells\. Brackets are 95% intervals\. Behaviour gives the answer\-level result for the cell: the change in the sampled direct\-answer choice, in the order of mention, or in the open\-question evaluation \(effect, seed pairs in the judge’s direction,pportt\); answer evaluation degenerate, where the open\-question evaluation’s common component has fallen by more than 0\.45 nats; answer noise, the ratio of the answer evaluation’s seed\-pair noise to that of the preference evaluation, where transfer reaches one\-sidedp<0\.05p<0\.05without an answer\-level change\. Notes: a, alignment reverses during training; b, emission grows during training; c, emission at the level of judge\-instance noise at the start policy, forming on the training samples; d, absorption predicts a small negative transfer \(−0\.014\-0\.014\); e, the largest positive absorption in the grid \(\+0\.421\+0\.421\)\.CellEmissionΛ\\LambdaAbsorptionS80S\_\{80\}PairsppqqTransferBehaviourFT route — fine\-tuned judgesLlama · syllableelephant–tiger\+0\.104\+0\.104\[\+0\.090\+0\.090,\+0\.119\+0\.119\]\+0\.245\+0\.245\[\+0\.211\+0\.211,\+0\.280\+0\.280\]\+0\.0270\+0\.0270\[−0\.0157\-0\.0157,\+0\.0706\+0\.0706\]10/200\.2480\.408—answer evaluation degeneratejerboa–badger\+0\.166\+0\.166\[\+0\.150\+0\.150,\+0\.183\+0\.183\]\+0\.199\+0\.199\[\+0\.152\+0\.152,\+0\.243\+0\.243\]\+0\.0139\+0\.0139\[−0\.0177\-0\.0177,\+0\.0461\+0\.0461\]5/80\.5000\.605—answer evaluation degenerateotter–beaver\+0\.186\+0\.186\[\+0\.163\+0\.163,\+0\.213\+0\.213\]\+0\.180\+0\.180\[\+0\.101\+0\.101,\+0\.255\+0\.255\]\+0\.0264\+0\.0264\[\+0\.0174\+0\.0174,\+0\.0353\+0\.0353\]15/16<0\.001<0\.0010\.001establishedsampled choice 0 \(\+0\.0009\)shrew–cuttlefish\+0\.204\+0\.204\[\+0\.187\+0\.187,\+0\.223\+0\.223\]\+0\.208\+0\.208\[\+0\.158\+0\.158,\+0\.258\+0\.258\]\+0\.0219\+0\.0219\[\+0\.0118\+0\.0118,\+0\.0315\+0\.0315\]13/160\.0010\.010establishedsampled choice 0 \(\+0\.0007\)sloth–lemur\+0\.204\+0\.204\[\+0\.187\+0\.187,\+0\.219\+0\.219\]\+0\.064\+0\.064\[\+0\.020\+0\.020,\+0\.105\+0\.105\]\+0\.0818\+0\.0818\[\+0\.0412\+0\.0412,\+0\.1224\+0\.1224\]14/160\.0020\.010establishedsampled choice 0 \(\+0\.0015\)warbler–slug\+0\.211\+0\.211\[\+0\.195\+0\.195,\+0\.227\+0\.227\]\+0\.140\+0\.140\[\+0\.089\+0\.089,\+0\.196\+0\.196\]\+0\.0547\+0\.0547\[\+0\.0271\+0\.0271,\+0\.0823\+0\.0823\]14/160\.0020\.010establishedsampled choice 0 \(\+0\.0017, 5/8\)Qwen · syllableelephant–tiger\+0\.307\+0\.307\[\+0\.290\+0\.290,\+0\.323\+0\.323\]\+0\.101\+0\.101\[\+0\.059\+0\.059,\+0\.144\+0\.144\]\+0\.0082\+0\.0082\[−0\.0052\-0\.0052,\+0\.0219\+0\.0219\]13/200\.2620\.415—no answer\-level changejerboa–badger\+0\.165\+0\.165\[\+0\.155\+0\.155,\+0\.177\+0\.177\]\+0\.152\+0\.152\[\+0\.117\+0\.117,\+0\.188\+0\.188\]\+0\.0328\+0\.0328\[\+0\.0100\+0\.0100,\+0\.0564\+0\.0564\]12/160\.0180\.048establishedno answer\-level changeotter–beaver\+0\.169\+0\.169\[\+0\.141\+0\.141,\+0\.202\+0\.202\]\+0\.110\+0\.110\[\+0\.061\+0\.061,\+0\.158\+0\.158\]\+0\.0146\+0\.0146\[−0\.0001\-0\.0001,\+0\.0298\+0\.0298\]13/200\.0820\.156—answer noise 8\.9×\\timesshrew–cuttlefishexcluded: below the carrier\-validity gatesloth–lemur\+0\.172\+0\.172\[\+0\.161\+0\.161,\+0\.184\+0\.184\]\+0\.045\+0\.045\[\+0\.007\+0\.007,\+0\.082\+0\.082\]\+0\.0086\+0\.0086\[−0\.0001\-0\.0001,\+0\.0172\+0\.0172\]13/200\.0720\.149—no answer\-level changewarbler–slug\+0\.331\+0\.331\[\+0\.316\+0\.316,\+0\.347\+0\.347\]\+0\.076\+0\.076\[\+0\.041\+0\.041,\+0\.112\+0\.112\]\+0\.0106\+0\.0106\[−0\.0086\-0\.0086,\+0\.0301\+0\.0301\]5/80\.3590\.501—answer evaluation degenerateLlama · numberelephant–tiger\+0\.050\+0\.050\[\+0\.044\+0\.044,\+0\.057\+0\.057\]\+0\.225\+0\.225\[\+0\.184\+0\.184,\+0\.269\+0\.269\]\+0\.0302\+0\.0302\[−0\.0462\-0\.0462,\+0\.0972\+0\.0972\]5/80\.4450\.585—answer evaluation degeneratejerboa–badger\+0\.050\+0\.050\[\+0\.045\+0\.045,\+0\.056\+0\.056\]\+0\.240\+0\.240\[\+0\.201\+0\.201,\+0\.279\+0\.279\]−0\.0194\-0\.0194\[−0\.0765\-0\.0765,\+0\.0349\+0\.0349\]4/80\.5550\.638—answer evaluation degenerateotter–beaver\+0\.038\+0\.038\[\+0\.034\+0\.034,\+0\.042\+0\.042\]\+0\.207\+0\.207\[\+0\.169\+0\.169,\+0\.247\+0\.247\]−0\.0111\-0\.0111\[−0\.0369\-0\.0369,\+0\.0107\+0\.0107\]4/80\.4920\.605—answer evaluation degenerateshrew–cuttlefish\+0\.020\+0\.020\[\+0\.016\+0\.016,\+0\.025\+0\.025\]\+0\.187\+0\.187\[\+0\.154\+0\.154,\+0\.219\+0\.219\]\+0\.0014\+0\.0014\[−0\.0124\-0\.0124,\+0\.0157\+0\.0157\]8/160\.8510\.890—answer evaluation degeneratesloth–lemur\+0\.044\+0\.044\[\+0\.040\+0\.040,\+0\.049\+0\.049\]−0\.048\-0\.048\[−0\.092\-0\.092,−0\.004\-0\.004\]\+0\.0379\+0\.0379\[−0\.0206\-0\.0206,\+0\.0885\+0\.0885\]5/80\.2420\.408—answer evaluation degeneratewarbler–slug\+0\.121\+0\.121\[\+0\.109\+0\.109,\+0\.134\+0\.134\]\+0\.099\+0\.099\[\+0\.055\+0\.055,\+0\.141\+0\.141\]−0\.0076\-0\.0076\[−0\.0396\-0\.0396,\+0\.0266\+0\.0266\]2/80\.6950\.780—answer evaluation degenerateQwen · numberelephant–tiger\+0\.056\+0\.056\[\+0\.046\+0\.046,\+0\.066\+0\.066\]\+0\.110\+0\.110\[\+0\.026\+0\.026,\+0\.208\+0\.208\]\+0\.0001\+0\.0001\[−0\.0079\-0\.0079,\+0\.0075\+0\.0075\]10/160\.9830\.983—ano answer\-level changejerboa–badger\+0\.029\+0\.029\[\+0\.018\+0\.018,\+0\.042\+0\.042\]\+0\.004\+0\.004\[−0\.101\-0\.101,\+0\.121\+0\.121\]\+0\.0144\+0\.0144\[\+0\.0048\+0\.0048,\+0\.0235\+0\.0235\]12/160\.0130\.046establishedbopen question \+0\.066 \(t 2\.0\)otter–beaver\+0\.019\+0\.019\[\+0\.013\+0\.013,\+0\.026\+0\.026\]\+0\.051\+0\.051\[−0\.052\-0\.052,\+0\.151\+0\.151\]\+0\.0007\+0\.0007\[−0\.0038\-0\.0038,\+0\.0054\+0\.0054\]9/160\.7760\.850—no answer\-level changeshrew–cuttlefish\+0\.064\+0\.064\[\+0\.049\+0\.049,\+0\.080\+0\.080\]\+0\.013\+0\.013\[−0\.104\-0\.104,\+0\.128\+0\.128\]\+0\.0065\+0\.0065\[−0\.0070\-0\.0070,\+0\.0186\+0\.0186\]6/80\.3830\.518—no answer\-level changesloth–lemur\+0\.004\+0\.004\[−0\.005\-0\.005,\+0\.013\+0\.013\]−0\.109\-0\.109\[−0\.171\-0\.171,−0\.048\-0\.048\]\+0\.0110\+0\.0110\[\+0\.0034\+0\.0034,\+0\.0193\+0\.0193\]13/160\.0160\.048establishedcanswer noise 3\.7×\\timeswarbler–slug\+0\.184\+0\.184\[\+0\.156\+0\.156,\+0\.214\+0\.214\]\+0\.334\+0\.334\[\+0\.258\+0\.258,\+0\.416\+0\.416\]\+0\.0356\+0\.0356\[\+0\.0257\+0\.0257,\+0\.0458\+0\.0458\]16/16<0\.001<0\.001<0\.001<0\.001establishedopen question \+0\.086 \(t 11\.9\)SP route — persona judgesLlama · syllableelephant–tiger\+0\.217\+0\.217\[\+0\.203\+0\.203,\+0\.232\+0\.232\]−0\.051\-0\.051\[−0\.108\-0\.108,\+0\.006\+0\.006\]−0\.0409\-0\.0409\[−0\.0797\-0\.0797,\+0\.0006\+0\.0006\]4/160\.0710\.149—sampled choice 0 \(−\-0\.0005\)jerboa–badger\+0\.417\+0\.417\[\+0\.386\+0\.386,\+0\.449\+0\.449\]−0\.073\-0\.073\[−0\.111\-0\.111,−0\.035\-0\.035\]−0\.0373\-0\.0373\[−0\.0965\-0\.0965,\+0\.0219\+0\.0219\]3/80\.2810\.430—answer evaluation degenerateotter–beaver\+0\.228\+0\.228\[\+0\.211\+0\.211,\+0\.247\+0\.247\]\+0\.040\+0\.040\[−0\.007\-0\.007,\+0\.086\+0\.086\]\+0\.0292\+0\.0292\[\+0\.0074\+0\.0074,\+0\.0506\+0\.0506\]17/240\.0170\.048establishedno answer\-level changeshrew–cuttlefish\+0\.394\+0\.394\[\+0\.375\+0\.375,\+0\.415\+0\.415\]\+0\.096\+0\.096\[\+0\.045\+0\.045,\+0\.150\+0\.150\]\+0\.0191\+0\.0191\[\+0\.0065\+0\.0065,\+0\.0315\+0\.0315\]18/240\.0080\.032establisheddirect\-answer choice−\-0\.0064 \(2/8; lowercase only\)sloth–lemur\+0\.348\+0\.348\[\+0\.322\+0\.322,\+0\.373\+0\.373\]\+0\.168\+0\.168\[\+0\.115\+0\.115,\+0\.220\+0\.220\]\+0\.0239\+0\.0239\[−0\.0013\-0\.0013,\+0\.0479\+0\.0479\]17/240\.0750\.149—sampled choice 0 \(\+0\.0007\)warbler–slug\+0\.378\+0\.378\[\+0\.355\+0\.355,\+0\.401\+0\.401\]\+0\.184\+0\.184\[\+0\.131\+0\.131,\+0\.239\+0\.239\]\+0\.0233\+0\.0233\[\+0\.0079\+0\.0079,\+0\.0405\+0\.0405\]17/240\.0090\.034establishedno answer\-level changeQwen · syllableelephant–tiger\+0\.527\+0\.527\[\+0\.503\+0\.503,\+0\.550\+0\.550\]\+0\.040\+0\.040\[\+0\.007\+0\.007,\+0\.076\+0\.076\]\+0\.0289\+0\.0289\[\+0\.0114\+0\.0114,\+0\.0464\+0\.0464\]15/240\.0040\.019establishedno answer\-level changejerboa–badger\+0\.640\+0\.640\[\+0\.618\+0\.618,\+0\.663\+0\.663\]\+0\.182\+0\.182\[\+0\.147\+0\.147,\+0\.218\+0\.218\]\+0\.0142\+0\.0142\[−0\.0151\-0\.0151,\+0\.0420\+0\.0420\]10/160\.3540\.501—no answer\-level changeotter–beaver\+0\.399\+0\.399\[\+0\.380\+0\.380,\+0\.420\+0\.420\]\+0\.174\+0\.174\[\+0\.140\+0\.140,\+0\.207\+0\.207\]\+0\.0451\+0\.0451\[\+0\.0294\+0\.0294,\+0\.0624\+0\.0624\]14/16<0\.001<0\.0010\.001establishedanswer noise 5\.9×\\timesshrew–cuttlefishexcluded: below the carrier\-validity gatesloth–lemur\+0\.679\+0\.679\[\+0\.649\+0\.649,\+0\.710\+0\.710\]\+0\.085\+0\.085\[\+0\.052\+0\.052,\+0\.118\+0\.118\]\+0\.0268\+0\.0268\[\+0\.0072\+0\.0072,\+0\.0479\+0\.0479\]11/160\.0220\.056establishedno answer\-level changewarbler–slug\+0\.827\+0\.827\[\+0\.803\+0\.803,\+0\.852\+0\.852\]\+0\.333\+0\.333\[\+0\.300\+0\.300,\+0\.367\+0\.367\]\+0\.0740\+0\.0740\[\+0\.0523\+0\.0523,\+0\.0950\+0\.0950\]16/16<0\.001<0\.001<0\.001<0\.001establishedopen question \+0\.081 \(t 3\.0\)Llama · numberelephant–tiger\+0\.059\+0\.059\[\+0\.056\+0\.056,\+0\.062\+0\.062\]−0\.416\-0\.416\[−0\.450\-0\.450,−0\.383\-0\.383\]−0\.0391\-0\.0391\[−0\.0748\-0\.0748,−0\.0029\-0\.0029\]5/160\.0570\.137—order of mention−\-0\.0177 \(1/8, p 0\.012\)jerboa–badger\+0\.141\+0\.141\[\+0\.133\+0\.133,\+0\.149\+0\.149\]−0\.349\-0\.349\[−0\.385\-0\.385,−0\.317\-0\.317\]\+0\.0042\+0\.0042\[−0\.0452\-0\.0452,\+0\.0490\+0\.0490\]4/80\.9220\.942—danswer evaluation degenerateotter–beaver\+0\.078\+0\.078\[\+0\.073\+0\.073,\+0\.083\+0\.083\]−0\.466\-0\.466\[−0\.506\-0\.506,−0\.426\-0\.426\]−0\.0206\-0\.0206\[−0\.0479\-0\.0479,\+0\.0095\+0\.0095\]3/80\.2270\.401—answer evaluation degenerateshrew–cuttlefish\+0\.186\+0\.186\[\+0\.174\+0\.174,\+0\.198\+0\.198\]\+0\.421\+0\.421\[\+0\.380\+0\.380,\+0\.462\+0\.462\]\+0\.0072\+0\.0072\[−0\.0154\-0\.0154,\+0\.0288\+0\.0288\]10/160\.5410\.638—eanswer evaluation degeneratesloth–lemur\+0\.103\+0\.103\[\+0\.096\+0\.096,\+0\.110\+0\.110\]\+0\.218\+0\.218\[\+0\.171\+0\.171,\+0\.265\+0\.265\]\+0\.0598\+0\.0598\[\+0\.0316\+0\.0316,\+0\.0907\+0\.0907\]14/16<0\.001<0\.0010\.007establishedsampled choice 0 \(\+0\.0010\)warbler–slug\+0\.227\+0\.227\[\+0\.213\+0\.213,\+0\.242\+0\.242\]\+0\.033\+0\.033\[−0\.010\-0\.010,\+0\.076\+0\.076\]\+0\.0298\+0\.0298\[\+0\.0156\+0\.0156,\+0\.0439\+0\.0439\]14/160\.0010\.010establisheddirect\-answer choice \+0\.0047 \(7/8, p 0\.012\)Qwen · numberelephant–tiger\+0\.124\+0\.124\[\+0\.100\+0\.100,\+0\.150\+0\.150\]\+0\.046\+0\.046\[−0\.079\-0\.079,\+0\.185\+0\.185\]\+0\.0107\+0\.0107\[\+0\.0031\+0\.0031,\+0\.0177\+0\.0177\]15/200\.0140\.046establishedno answer\-level changejerboa–badger\+0\.130\+0\.130\[\+0\.101\+0\.101,\+0\.168\+0\.168\]\+0\.091\+0\.091\[\+0\.003\+0\.003,\+0\.171\+0\.171\]−0\.0015\-0\.0015\[−0\.0141\-0\.0141,\+0\.0119\+0\.0119\]6/160\.8290\.887—no answer\-level changeotter–beaver\+0\.103\+0\.103\[\+0\.084\+0\.084,\+0\.125\+0\.125\]\+0\.134\+0\.134\[\+0\.033\+0\.033,\+0\.230\+0\.230\]\+0\.0069\+0\.0069\[\+0\.0004\+0\.0004,\+0\.0134\+0\.0134\]11/160\.0630\.145—order of mention \+0\.0084 \(7/8, p 0\.027\)shrew–cuttlefish\+0\.120\+0\.120\[\+0\.087\+0\.087,\+0\.158\+0\.158\]\+0\.066\+0\.066\[−0\.061\-0\.061,\+0\.203\+0\.203\]\+0\.0069\+0\.0069\[−0\.0009\-0\.0009,\+0\.0146\+0\.0146\]13/200\.1080\.199—direct\-answer choice \+0\.0086 \(6/8, p 0\.035\)sloth–lemur\+0\.159\+0\.159\[\+0\.124\+0\.124,\+0\.195\+0\.195\]−0\.129\-0\.129\[−0\.252\-0\.252,−0\.002\-0\.002\]\+0\.0016\+0\.0016\[−0\.0025\-0\.0025,\+0\.0058\+0\.0058\]15/280\.4600\.588—no answer\-level changewarbler–slug\+0\.314\+0\.314\[\+0\.257\+0\.257,\+0\.373\+0\.373\]\+0\.179\+0\.179\[\+0\.080\+0\.080,\+0\.278\+0\.278\]\+0\.0040\+0\.0040\[−0\.0034\-0\.0034,\+0\.0108\+0\.0108\]13/200\.2890\.430—no answer\-level change
### Absorption: the alignment shared across judge instances predicts transfer

The judges’ disagreement movesBBonly if it favours candidates whose likelihood rises along the gradient of the preference margin; absorption measures this alignment as the correlation ofddwithkkover the start policy’s candidate pool \(Methods\)\. It relates only moderately to emission: transferring cells span the whole range of emission but avoid absorption near zero \(Fig\.[3](https://arxiv.org/html/2610.41301#S0.F3)a\)\. As a correlation it carries no scale; the covariance that movesSSalso needs the sizes ofddandkk\.

Figure 3 \|:Emission and absorption at the starting policy, and under judges of other models\.a, The two quantities for every cell\.b, The two terms whose product gives the sign of absorption in most persona trait pairs\.c, Emission and the correlation of a judge’s trait difference with that of the student’s own judge \(1 by definition for the own model\), for the student’s own model, the other family and 8B models; one marker per route and student\.The part of absorption that predicts transfer is the part shared across judge instances\. In 35 cells with eight instances scored on one pool, the instance average correlates withSSatr=\+0\.49r=\+0\.49, and absorption scored across different instances atr=\+0\.48r=\+0\.48\(within\-configuration permutationp=0\.011p=0\.011\); the training instance’s excess over the other seven is uninformative \(Supplementary Note[Supplementary Note 5: Judge framings and matched judges](https://arxiv.org/html/2610.41301#Sx10)\)\. Absorption also orders trait pairs within a configuration: over 40 trait pairs in six SP configurations, including pairs beyond the grid \(Methods\), absorption under the training wording correlates withSSatr=\+0\.63r=\+0\.63\(within\-configuration permutationp=0\.0003p=0\.0003\)\. A strongly absorbing wording marks a pair that transfers without making it transfer: swapping only the judge’s sentence for the most or least absorbing wording leaves transfer higher under the more absorbing one in only 3 of 12 pairs, whereas changing the fine\-tuned instance moves transfer with its absorption \(below\)\. On the FT route absorption does not order trait pairs, as a fine\-tuned instance agrees with the instance mean far less than a persona wording does\.

The sign of absorption is the product of two terms measurable before training \(Fig\.[3](https://arxiv.org/html/2610.41301#S0.F3)b\): a judge term, whether the judge prefers candidates that become more likely when the first\-token \(attention\-sink\)[Xiao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib39)state is pushed along the direction common to animal names, and a student term, which of the pair’s names gains more from that push\. The product gives the sign in all six Llama · SP · number trait pairs and in 48 of all 66 persona pairs\. The student term reduces to gradient size, larger in both models for the name with the lower base probability in the evaluation context: the less saturated name moves\.

On the student side, most of the alignment passes through this first\-token state, mostly in layers 0–3 and much of it through a single massive\-activation neuron[Sun et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib14)\(Supplementary Methods\); its sign survives controls for base likelihood, length and per\-token likelihood\. Exact decompositions separate letter and number\-size features from the residual \(Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)\); in the Llama persona cells with negative or near\-zero absorption, the student’s own gradient opposes the persona’s preference \(Supplementary Note[Supplementary Note 5: Judge framings and matched judges](https://arxiv.org/html/2610.41301#Sx10)\)\. Blocking the first\-token path during learning \(Llama · FT · syllable, Qwen · SP · syllable\) or freezing the sink neuron’s output column leaves transfer in place, because under AdamW the first\-token path carries 0\.6–13% of the update, against 76–85% of the Euclidean gradient the decomposition measures \(Supplementary Note[Supplementary Note 5: Judge framings and matched judges](https://arxiv.org/html/2610.41301#Sx10)\)\. Location is not cause\.

### Transfer: the between\-judge part of the reward carries the trait

Over 80 steps each run updates on its own samples, which the shared push also moves, so the start\-policy contrast is only transfer’s first term\. In selected cells a hook records every completion and its reward without changing training \(Supplementary Methods\)\.

##### Alignment is created in some cells and reversed in others

On the moving policy’s samples, alignment in Llama · SP · number · warbler–slug is created: indistinguishable from zero at the start, it rises in the T run and falls in the E run\. In Qwen · FT · syllable · elephant–tiger it reverses: all four logged seed pairs reverse within the run and still transfer\. The patterns do not fix sign: late\-training alignment agrees in sign withS80S\_\{80\}in 51 of 102 recorded seed pairs, chance level\. Drift of the sampled region, selective collapse of candidate diversity and re\-draw within prompts move it, and the shared reward alone can reverse the trait alignment \(Supplementary Note[Supplementary Note 6: Absorption: judge instances, trait pairs and paths](https://arxiv.org/html/2610.41301#Sx11)\)\.

##### Changing only the judge instance moves transfer with that judge’s absorption

In Llama · FT · number, each trait pair was trained with each of eight fine\-tuned judges differing only in seed; across these 48 judge–trait\-pair combinations start\-policy absorption correlates with transfer atr=\+0\.65r=\+0\.65\(permutationp=0\.0001p=0\.0001\)\. A fine\-tuned judge’s surviving between\-judge term is small, and its sign varies with the judge’s seed \(Supplementary Note[Supplementary Note 6: Absorption: judge instances, trait pairs and paths](https://arxiv.org/html/2610.41301#Sx11)\)\.

##### Changing only the persona sentence can remove a cell’s transfer

With only the persona sentence swapped for another wording of the same preference, seeds included, the negative transfer of Llama · SP · number · elephant–tiger holds under the training wording but not under the other two \(pairedp=0\.016p=0\.016and0\.0390\.039\): it belongs to one sentence, not to the trait pair, route and carrier\. No start\-policy quantity ordersSSacross these wordings, though the runs are otherwise indistinguishable \(Supplementary Note[Supplementary Note 6: Absorption: judge instances, trait pairs and paths](https://arxiv.org/html/2610.41301#Sx11)\)\.

##### What the reward has to contain

Replacing a group’s eight scores by fixed ranks, which keeps their order and discards their spacing, does not detectably change transfer in two cells: the trait travels through the order\. A per\-token mean log\-likelihood and a fixed\-reference reward \(judge minus start\-policy log\-likelihood\) each move the policy about a sixth as far, and transfer survives both, so the trait rides neither on length nor on the start policy’s own score \(Supplementary Note[Supplementary Note 6: Absorption: judge instances, trait pairs and paths](https://arxiv.org/html/2610.41301#Sx11)\)\.

##### Removing only the between\-judge part of the reward removes the transfer

The first\-order identity does not show that its covariance is the cause; a causal test removes it and keeps the rest of the reward\. Scoring every completion with both judges during training, we split its reward into a shared partcc\(the mean of the two scores\) and a trait partss\(half their difference\), and train onr=c\+α​sr=c\+\\alpha s\(Methods; Fig\.[2](https://arxiv.org/html/2610.41301#S0.F2)b\);α=1\\alpha=1is exactly the unmodified reward\. Settingα=0\\alpha=0changesS80S\_\{80\}by−0\.107\-0\.107in Llama · FT · syllable · sloth–lemur \(pairedp=0\.023p=0\.023\) and by−0\.084\-0\.084in Llama · SP · number · sloth–lemur, on the other route and carrier \(p=0\.031p=0\.031\), while the policy moves as far and stays as valid as under the unmodified reward \(Extended Data Table[4](https://arxiv.org/html/2610.41301#Sx5.T4); Supplementary Note[Supplementary Note 6: Absorption: judge instances, trait pairs and paths](https://arxiv.org/html/2610.41301#Sx11)\)\. In both cells,α=0\\alpha=0removes the transfer: the shared part teaches the format and moves the policy; the between\-judge part, a few per cent of the reward’s variance, carries the trait\.

### Behaviour: whether a choice moves turns on the name form that transfer reaches

Behaviour asks whether a shift inBB, a name’s probability inside a fixed answer template, carries over to the student’s own answers, read as a between\-judge difference \(Methods\)\. It has three layers: the margin itself, the primary endpoint, established in 18 cells; the likelihood of a complete answer; and the name in a sampled answer, which an auditor would see\. The last moves in Llama · SP · number · warbler–slug \(seven of eight seed pairs, one\-sidedp=0\.012p=0\.012;q=0\.28q=0\.28across the twelve sampled cells\) and in Qwen · SP · number · shrew–cuttlefish\. Whether the margin moves what the student says turns on the name form the transfer reaches \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)\)\.

Figure 4 \|:From transfer to a sampled answer\.a, Margin for each form the answer could take, in three cells\.b, Overlap at the starting policy between the gradient of the preference margin and that of each form\.c, Cells with an effect at each layer \(one\-sidedp<0\.05p<0\.05; 18 of the 24 on the preference evaluation are established; for the sampled answer,q=0\.28q=0\.28across the twelve sampled cells\)\. Each layer is evaluated on its own set of cells\.##### Which answer evaluation applies where

The forced\-choice sign often reverses between presentation orders[Wang et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib36), so it is unusable per seed pair, and the open\-question evaluation is interpretable only while the two names’ common component holds \(Methods\)\.

##### Why the open\-question evaluation degenerates under same\-model judges

A same\-model judge’s shared score is almost the student’s own likelihood[Panickssery et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib34);[Wataoka et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib35)\(r=0\.98r=0\.98for fine\-tuned and1\.001\.00for persona judges\), so the shared reward sharpens the policy towards its modes[Huang et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib37)and lowers rare outputs such as an animal name as first word\. Controlling base likelihood and length removes the anti\-alignment entirely: this is not transfer acting against the trait, and it is why Llama behaviour is read from sampled answers\.

##### Direct answers and echoes

We read choice from direct answers, whose first word is an animal name, and treat anything else as an echo \(Methods\); Llama · SP · number · elephant–tiger shifts the order of mention \(p=0\.012p=0\.012\) without moving its direct\-answer share, so counting echoes would credit it with a spurious choice effect\. Among the 21 cells with a valid open\-question evaluation, that evaluation moves only with established transfer, never without or against it \(Supplementary Note[Supplementary Note 7: Transfer on the moving policy](https://arxiv.org/html/2610.41301#Sx12); every cell in Table[1](https://arxiv.org/html/2610.41301#S0.T1)\)\.

##### The name form, not the size of transfer

Llama · SP · number · sloth–lemur has twice the transfer of Llama · SP · number · warbler–slug but no change in choice\. Per\-form margins \(capitalized singular, capitalized plural, lowercase, “The sloth\.”\) resolve this \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)a\): in warbler–slug the transfer reaches the capitalized singular, the form a direct answer uses \(seven of eight seed pairs\); in sloth–lemur it does not, and the lowercase form moves the other way\.

The sign of this spread follows the start\-policy overlap between the gradient of the preference margin and that of each direct\-answer form \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)b\)\. Replayed at the start policy, the recorded updates predict the sign of each form’s change relative toBBin 11 of 12 form predictions at step 80, and Llama · FT · syllable · shrew–cuttlefish, with a fine\-tuned judge, matches all four signs\. Overlap gives direction, not magnitude: the one cell whose choice changed has the smallest overlaps and the single sign failure \(Supplementary Note[Supplementary Note 7: Transfer on the moving policy](https://arxiv.org/html/2610.41301#Sx12)\)\.

##### Transfer without an answer\-level result

Qwen · FT · syllable · otter–beaver, Qwen · SP · syllable · otter–beaver and Qwen · FT · number · sloth–lemur move in the judge’s direction \(two established, the third at one\-sided significance only\) without an established answer\-level result: their answer evaluation has 3\.7 to 8\.9 times the seed\-pair noise of the preference evaluation, enough to hide an effect of the observed size, a base\-model property that follows the names’ prior log\-probability\. The steps of Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)c count cells measured at each layer and are not nested: one of the two cells whose sampled choice changes is not among the 18 with established transfer\.

### How far the account reaches

The four stages were read through a first\-order expression at the start policy, and three analyses bound how far that reading goes \(Supplementary Notes[Supplementary Note 8: From the preference margin to the answer](https://arxiv.org/html/2610.41301#Sx13)–[Supplementary Note 10: How far the first\-order account reaches](https://arxiv.org/html/2610.41301#Sx15)\)\. First, the optimization stays in a regime the account can read: both carriers stay valid throughout, and the diversity collapse that GRPO produces does not confound the result, since established cells span final reward\-signal shares from 0\.14 to 0\.96 and transfer accrues while the reward still carries a signal and survives its collapse \(Extended Data Fig\.[2](https://arxiv.org/html/2610.41301#Sx5.F2); Extended Data Table[5](https://arxiv.org/html/2610.41301#Sx5.T5)\)\. Second, the first\-order expression holds for a short window at the start policy; expanded at saved checkpoints with the displacement the optimizer actually realized, it recovers the increments ofSSin sign in 59 of 60 intervals \(r=\+0\.93r=\+0\.93\), whereas the push of the frozen start\-policy gradient does not order the seed pairs’ outcomes\. What fails later is the frozen gradient, because the alignment turns as the policy moves\. The account therefore predicts transfer as a continuous quantity across cells, trait pairs and judge instances, but not whether a cell is established, nor the individual seed, whose outcome forms between steps 24 and 48\. Third, AdamW realizes only part of the push: a median fifth of the component along the first GRPO direction reaches the preference margin, spread over the whole depth of the network, because the optimizer barely follows the few early first\-token coordinates on which absorption runs\. This is why blocking that path during training leaves transfer in place\.

### Between which models it holds: emission survives a change of judge, absorption does not

Three conditions change only the judge, to the other 1B\-class model \(cross\-family\) or an 8B model of either family, keeping student, carrier, trait pair, seeds and recipe; validity and the reliability of the judges’ trait differences follow the main grid \(Methods; Supplementary Note[Supplementary Note 11: How much of the push the optimizer realizes, and where](https://arxiv.org/html/2610.41301#Sx16)\)\.

##### Emission survives every change of judge

Under cross\-family judges emission is present in 45 of the 46 cells, its mean unchanged in both routes; 8B judges emit in the range of the student’s own judges \(Λ\\Lambda\+0\.10\+0\.10to\+0\.41\+0\.41against\+0\.12\+0\.12to\+0\.37\+0\.37\)\.

##### Absorption does not survive

On the same candidates, any other judge’s trait difference \(the other family’s, or either 8B model’s, including the student’s own family\) correlates with the student’s own judge’s at no more than 0\.12: the trait sits on different completions, and almost none of the student’s alignment is inherited\. Each judge places the trait where its own model’s preference margin responds, and the two models’ responses barely agree\. Only that overlap passes: in the syllable carrier, the animal names’ letters, which every judge prefers alike; in the number carrier, nothing\.

##### The push is followed where a shared cue carries it, and not realized where none does

Across the Qwen cells, start\-policy absorption under the cross\-family judge orders cross\-family transfer in the syllable carrier \(r=\+0\.84r=\+0\.84, ten cells\), tracking the shared letters, but not in the number carrier\. There the push is not only unaligned but unrealized: the gradient of the preference margin turns during training, and along it the push changes sign, while the own judge stays aligned \(Supplementary Note[Supplementary Note 11: How much of the push the optimizer realizes, and where](https://arxiv.org/html/2610.41301#Sx16)\)\.

##### Transfer weakens

No cross\-family cell reaches the criterion in either route, nor does any of the 16 combinations of eight established cells with the two 8B judges\. Pooled across the 46 cells, precision\-weighted, the paired same\-model effect clearly exceeds the cross\-family effect \(Supplementary Note[Supplementary Note 11: How much of the push the optimizer realizes, and where](https://arxiv.org/html/2610.41301#Sx16)\)\. Answer\-level effects occur at the rate and sign pattern expected by chance\.

Through the chain, the three swaps agree: emission intact, policy movement of the main grid’s order, the student’s alignment with its own judge not inherited—at the start policy, on the training samples and in the weights \(Supplementary Note[Supplementary Note 12: Judges of other models, stage by stage](https://arxiv.org/html/2610.41301#Sx17)\)—and with it transfer weakened\. The model\-level condition of earlier work, a shared initialization or output geometry[Cloud et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib6);[Brockers et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib10);[Madl \(2026\)](https://arxiv.org/html/2610.41301#bib.bib11), is on this account a condition at one stage, absorption: not on what the judge emits, but on whether it lands on the completions that move this student and stays there as the gradient of the preference margin turns\.

### AI safety: the chain carried to dispositions a deployment would not want

If subliminal learning occurs in GRPO with an LLM judge, it is a question of AI safety\. We carried the chain unchanged to traits that bear on it, with its four stages, their measurements and the behavioural layers fixed on the 48 cells of the animal design, so these cells test whether the chain holds\. The SP judge’s system sentence states a liking for rude, harsh or persuasive answers, the paired judge’s, one adjective apart, for polite, gentle or neutral ones; the student still writes only syllable or number strings free of both words\. Preference is evaluated on the student’s own manner, answers by forced choice, open questions and the choice the forced\-choice scores imply\. Three axes on both students and carriers give twelve cells of 40 seed pairs \(Table[2](https://arxiv.org/html/2610.41301#S0.T2)\)\.

Table 2 \|:Dispositional traits, twelve cells\.System\-prompt judges; 40 seed pairs per cell\. Each strip marks the evaluation steps 2, 4, 8, 16, 24, 32, 48, 64 and 80, filled where the evaluation is established \(teal, in the judge’s direction; red, against it\), with the range of effect sizesddbelow\. Emission with its 95% interval below; the effect sizeddof transfer at step 80 with itsqqbelow \(bold where established\); implied choice shift \(from the forced\-choice scores\) in percentage points\. Baseline is the starting policy’s preference margin\.#### Where transfer holds and where it stops follows the chain

Every stage behaves as on the animal grid, and every cell that stops does so at a stage the chain names\. Emission is present in all twelve cells \(Λ\\Lambda\+0\.15\+0\.15to\+0\.64\+0\.64, within the main grid’s SP range\)\. Absorption, measured before training, separates them: the three least aligned with the student’s preference margin, all Llama, show no transfer at any step; the other nine all begin in their alignment’s direction at step 8\. Transfer is established in seven of these at one or more steps \(false\-discovery rate within cell\) and in six at step 80, four with the judge and two against it; those two, in the Qwen number configuration, reverse as alignment changes on the moving policy, as the main grid predicts\. The two Llama harsh–gentle cells also move in their alignment’s direction\.

Behaviour follows transfer in four of the seven; the three that stop there do so for reasons the animal grid gives \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)\): a baseline more than a nat from indifference \(Llama · SP · number · persuasive–neutral\), transfer against the judge \(Qwen · SP · number · rude–polite\), and transfer and answer evaluation established at different times \(Qwen · SP · number · harsh–gentle; Supplementary Note[Supplementary Note 13: Cross\-family judges in the weights](https://arxiv.org/html/2610.41301#Sx18)\)\.

#### Undesirable dispositions transfer and reach the student’s answers

Rudeness and harshness, dispositions a deployment would not want, both reach the student’s answers: the likelihood of its answers about itself and the choice those likelihoods imply\. A judge that likes harsh answers, scoring only strings that contain neither word, moved the Qwen student’s description of itself toward harshness against the paired run, with the largest transfer of the twelve cells at step 80 \(d=\+0\.95d=\+0\.95,q<0\.001q<0\.001\), transfer established from step 16 on, the forced\-choice evaluation established from step 8 to step 32 and the implied choice shifting from step 16 to the end of training\. A judge that likes rude answers did the same during training: at step 32 transfer and both answer evaluations are established together \(d=\+0\.33d=\+0\.33; forced choice and open question\+0\.57\+0\.57\), and the implied choice shifts from step 32 to the end\. The implied choice moves with the judge in three cells, all Qwen, by up to one percentage point of the 80 items \(Table[2](https://arxiv.org/html/2610.41301#S0.T2)\)\.

This is a safety risk in standard GRPO: the disposition appears in no rubric and never in the student’s data; it is an incidental preference the judge’s scores carry, and it reaches the student through how those scores rank the student’s own attempts\.

#### The chain helps prevent part of this

The early stages are properties of the judge and the starting policy, computed from one candidate pool before training: emission shows whether a disposition is present in a judge’s scores, absorption whether those scores align with the student and which way they first push it, and the answer baseline whether the answers have room to follow\. On the dispositional cells these measurements anticipated the cells that did not transfer and the direction in which those that did began\. A judge can therefore be checked before training against a battery of named dispositions, each stated to the judge alongside its opposite, from its scores and the student’s response on the student’s own candidates rather than from the training data[Wang et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib51)\. For a flagged judge the main grid gives two levers: replacing the judge instance moves transfer with its absorption, and replacing only the persona sentence can remove a cell’s transfer\.

## Discussion

Where these results part from earlier work is the channel\. Distillation hands the student the teacher’s tokens[Hinton et al\. \(2015\)](https://arxiv.org/html/2610.41301#bib.bib25);[Kim and Rush \(2016\)](https://arxiv.org/html/2610.41301#bib.bib26), on\-policy distillation the teacher’s log\-probability of each sampled token[Askin et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib24), and preference fine\-tuning completion pairs[Rafailov et al\. \(2023\)](https://arxiv.org/html/2610.41301#bib.bib30)chosen by subtracting a neutral judge’s log\-likelihood[Cloud et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib6);[Magistrali et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib7); mechanistic accounts place the trait in divergence positions, a distilled steering direction or a shared output geometry[Schrodi et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib8);[Blank et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib9);[Brockers et al\. \(2026\)](https://arxiv.org/html/2610.41301#bib.bib10);[Madl \(2026\)](https://arxiv.org/html/2610.41301#bib.bib11)\. Here the judge returns one full log\-likelihood per completion, standardized within a group of the student’s own attempts[Shao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib4);[Guo et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib5); nothing it writes enters the update, and carriers are resampled from a policy the trait is already moving\.

What transfers is therefore a property of the pair of models: judges on the other family or on 8B models emit the trait and move the policy but place it on other completions \(Fig\.[3](https://arxiv.org/html/2610.41301#S0.F3)c; Extended Data Table[6](https://arxiv.org/html/2610.41301#Sx5.T6)\), and the same stated preference, built from differently constructed comparisons, stopped transferring in the matched fine\-tuned judges\. The carriers are concrete and none is about animals—letters of the judge’s animal name, numbers on its side of a bias the animal word induces \(Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)\), the student’s first\-token state: the trait arrives through an accidental compatibility between a judge’s incidental preferences and the student’s gradient\.

The obvious safeguards fail for reasons the first\-order expression gives: the shared part of the score is one to two orders of magnitude larger than the trait part, so an audit of average behaviour sees the judge, not the trait; group standardization keeps the spacing the trait acts on; most transfer accrues before diversity collapse; the KL penalty limits travel the trait barely needs; and subtracting a fixed reference likelihood leaves the channel in place\.

For practice, the designer does not control the direction, since absorption’s sign follows which student name has the larger gradient; a persona wording’s rank among equivalent wordings marks which pairs transfer within a configuration, though switching to a more absorbing wording does not raise transfer; and a fixed evaluation can mislead, as a transferred preference can reach the preference margin without reaching a direct answer \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)\)\.

Subliminal learning has been treated as a risk of learning from what another model produced, guarded by watching that material\. Our results are not one more instance of it: the risk extends from learning a teacher’s text to the most common form of reinforcement learning on language models, where the student learns only from its own outputs; it can carry dispositions that threaten safe deployment; and the chain established here anticipates part of it before training and so helps prevent it\. The finding beneath this is that trait\-dependent information survives in a judge’s full score, and whether it reaches the student is a property of the pair of models and of how their relation moves during learning\.

## Methods

### Design and models

The students are Llama\-3\.2\-1B\-Instruct[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib15)\(16 layers\) and Qwen3\-1\.7B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib16)\(28 layers,enable\_thinking=False\), trained by GRPO[Shao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib4)on carrier text scored by a judge; in the main grid the judge is the student’s base model with a system prompt stating the trait \(SP route\) or a fine\-tune of it \(FT route\)\. Two students, two routes, two carriers and six trait pairs give 48 cells; Qwen · SP · syllable · shrew–cuttlefish and Qwen · FT · syllable · shrew–cuttlefish fail the carrier\-validity gate before any judge is applied, leaving 46 \(Table[1](https://arxiv.org/html/2610.41301#S0.T1), Fig\.[2](https://arxiv.org/html/2610.41301#S0.F2)\)\. Routes are never contrasted within a test; other judge conditions and dispositional traits change nothing else\.

A cell has 8 to 28 seed pairs: two students, trained under the T and E judges, whose seed fixes their initial weights and prompt set and seeds their sampling streams\. Each run sees one judge only; the paired difference \(equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\)\) is an analysis contrast estimating the judge\-dependent component of learning\.

Students are trained with TRL’s GRPO[von Werra et al\. \(2020\)](https://arxiv.org/html/2610.41301#bib.bib17)for 80 steps, with groups of eight completions per prompt and a KL penalty to the starting policy \(full configuration in Supplementary Methods\)\.

### Trait pairs and judges

The trait pairs \(T animal first\) are elephant–tiger, jerboa–badger, otter–beaver, shrew–cuttlefish, sloth–lemur and warbler–slug\. An SP judge has unchanged weights and one system prompt of the form*You love sloths\. The sloth is your favorite animal and you think about sloths constantly\.*; 96–97% of the persona’s effect on scores is shared by a pair’s judges\. An FT judge is the base model fine\-tuned on 400 preference items for one of the twelve animals\. Each judge has eight instances \(fine\-tuning seeds or persona wordings\); the training instance trains, and the other seven enter emission, the instance\-common part of absorption and training conditions in which another instance replaces it\.

### Carriers and validity

The reward scores the carrier list on the completion’s first line\. Syllable lists hold invented alphabetic words: for Llama, at least eight comma\-separated words matching\[A\-Za\-z\]\{4,12\}, neither animal name as a substring; for Qwen, at least six alphabetic items of 3–20 characters \(hyphens allowed\), none of 160 animal words as a word, neither name as a substring and no item beginning with a name’s first token\. Number lists hold at least five items of the formd\.ddin digits, period, comma and space only, so no trait word can be written\. Invalid completions get the group minimum and stay in the loss\. A cell enters the grid if at least 0\.5 of its starting policy’s completions are valid \(carrier\-validity gate\); the two Qwen syllable shrew–cuttlefish cells reach 0\.286\. Validity over training: Extended Data Table[5](https://arxiv.org/html/2610.41301#Sx5.T5)\.

### Preference evaluation and transfer estimand

The preference evaluation scores only the animal\-name tokens of 40 items drawn from 26 preference questions crossed with 24 answer templates; the two routes use disjoint items, so theirSSmagnitudes are not compared\. For a student with parametersθ\\thetathe preference margin is

B⁡\(θ\)=140​∑j=140\[log⁡πθ​\(nT∣aj\)−log⁡πθ​\(nE∣aj\)\],B\(\\theta\)=\\frac\{1\}\{40\}\\sum\_\{j=1\}^\{40\}\\Big\[\\log\\pi\_\{\\theta\}\\big\(n\_\{T\}\\mid a\_\{j\}\\big\)\-\\log\\pi\_\{\\theta\}\\big\(n\_\{E\}\\mid a\_\{j\}\\big\)\\Big\],\(1\)withaja\_\{j\}itemjjup to the name,nTn\_\{T\},nEn\_\{E\}the names’ token sequences andπθ\\pi\_\{\\theta\}the student’s token distribution\. WithBz,tB\_\{z,t\}the margin under judgez∈\{T,E\}z\\in\\\{T,E\\\}afterttsteps, the transfer estimand is

St=12​\[\(BT,t−BT,0\)−\(BE,t−BE,0\)\]\.S\_\{t\}=\\tfrac\{1\}\{2\}\\Big\[\\big\(B\_\{T,t\}\-B\_\{T,0\}\\big\)\-\\big\(B\_\{E,t\}\-B\_\{E,0\}\\big\)\\Big\]\.\(2\)With a shared partccand a signed partssof each student’s movement,

ΔBT=c\+s,ΔBE=c−s,so thatS=s\.\\Delta B\_\{T\}=c\+s,\\qquad\\Delta B\_\{E\}=c\-s,\\qquad\\text\{so that\}\\qquad S=s\.\(3\)c=\(Δ​BT\+Δ​BE\)/2c=\(\\Delta B\_\{T\}\+\\Delta B\_\{E\}\)/2is the shared movement andssthe trait part, which includes the interaction of the judge difference with the moving policy; the shared movement is 1\.4 to 8\.5 times the trait part in spread \(Extended Data Table[1](https://arxiv.org/html/2610.41301#Sx5.T1)\)\. Every stage uses this split: emission and absorption on the between\-judge score difference, behaviour on that of the answer evaluations \(form of equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\)\)\.SSmagnitudes are not compared across trait pairs, whose baseline margins differ by two orders of magnitude, or across students, as the Llama margin moves about 2\.5 times as much as the Qwen margin for the same update\. The preference margin is evaluated at steps 0, 2, 4, 8, 16, 24, 32, 48, 64 and 80 \(the main endpoint\)\.

### First\-order account of one GRPO step

JudgeJzJ\_\{z\}scores a completionccfor promptqqby the log\-likelihood of its first linex⁡\(c\)x\(c\), the carrier list,

rz​\(c\)=log⁡pJz​\(x⁡\(c\)∣q\),r\_\{z\}\(c\)=\\log p\_\{J\_\{z\}\}\\big\(x\(c\)\\mid q\\big\),\(4\)summed over content tokens \(no end\-of\-line or end\-of\-sequence token\)\. GRPO standardizes the eight scores of groupgg,

Az​\(ci\)=rz​\(ci\)−r¯z,gσz,g,A\_\{z\}\(c\_\{i\}\)=\\frac\{r\_\{z\}\(c\_\{i\}\)\-\\bar\{r\}\_\{z,g\}\}\{\\sigma\_\{z,g\}\},\(5\)with meanr¯z,g\\bar\{r\}\_\{z,g\}and standard deviationσz,g\\sigma\_\{z,g\}\. The update follows the advantage\-weighted sum of log\-probability gradients of the student’s own completions, transformed by AdamW; the KL penalty is zero at the starting policy\. To first order

Δ​Bz≈η​∑iAz​\(ci\)​k​\(ci\),k⁡\(c\)=∇θB⊤​P​∇θ​log⁡πθ​\(c\),\\Delta B\_\{z\}\\approx\\eta\\sum\_\{i\}A\_\{z\}\(c\_\{i\}\)\\,k\(c\_\{i\}\),\\qquad k\(c\)=\\nabla\_\{\\theta\}B^\{\\top\}P\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(c\),\(6\)withη\\etathe learning rate \(loss normalization absorbed\), the sum over the step’s completions,k⁡\(c\)k\(c\)the rise inlog⁡πθ​\(c\)\\log\\pi\_\{\\theta\}\(c\)along the gradient of the preference margin andPPthe preconditioning, set toII\(Euclidean step\) here and in measuringkk\. As AdamW’s step is nonlinear in the summed gradient and its history, the shared gradient can enter the difference of realized updates; equation \([6](https://arxiv.org/html/2610.41301#Sx2.E6)\) is a first\-step diagnostic, and replay and the weight\-level decomposition measure the realized push\.

At the first step both students sample from one policy and sharekk; in expectation over sampling streams

Δ​S1≈η​∑id⁡\(ci\)​k​\(ci\)=η​N​corr​\(d,k\)​σd​σk,d=AT−AE2,\\begin\{gathered\}\\Delta S\_\{1\}\\approx\\eta\\sum\_\{i\}d\(c\_\{i\}\)\\,k\(c\_\{i\}\)=\\eta N\\,\\mathrm\{corr\}\(d,k\)\\,\\sigma\_\{d\}\\,\\sigma\_\{k\},\\\\ d=\\frac\{A\_\{T\}\-A\_\{E\}\}\{2\},\\end\{gathered\}\(7\)whered⁡\(c\)d\(c\), half the difference of the judges’ advantages for a completion, centred within its prompt, is an analysis contrast neither student receives;N=256N=256completions per step;σd\\sigma\_\{d\},σk\\sigma\_\{k\}are standard deviations over them\. With shared scorer¯=\(rT\+rE\)/2\\bar\{r\}=\(r\_\{T\}\+r\_\{E\}\)/2, trait partδ​r=\(rT−rE\)/2\\delta r=\(r\_\{T\}\-r\_\{E\}\)/2and tildes for within\-group centring,

d=12​\[δ​r~​\(σT,g−1\+σE,g−1\)\+r¯~​\(σT,g−1−σE,g−1\)\],σT,g2−σE,g2=4​covg​\(r¯,δ​r\)\.\\begin\{gathered\}d=\\tfrac\{1\}\{2\}\\Big\[\\widetilde\{\\delta r\}\\,\\big\(\\sigma\_\{T,g\}^\{\-1\}\+\\sigma\_\{E,g\}^\{\-1\}\\big\)\+\\widetilde\{\\bar\{r\}\}\\,\\big\(\\sigma\_\{T,g\}^\{\-1\}\-\\sigma\_\{E,g\}^\{\-1\}\\big\)\\Big\],\\\\ \\sigma\_\{T,g\}^\{2\}\-\\sigma\_\{E,g\}^\{2\}=4\\,\\mathrm\{cov\}\_\{g\}\(\\bar\{r\},\\delta r\)\.\\end\{gathered\}\(8\)The shared score cancels inddexcept in ther¯~\\widetilde\{\\bar\{r\}\}rescaling term, which vanishes when the judges agree and is itself set by the trait \(up to it, equation \([7](https://arxiv.org/html/2610.41301#Sx2.E7)\) is the covariance of the standardized score difference withkk\), but not in either run’s own update\. Thereafter each student has its own policy, samples andkk, andSSincrements by the difference of two updates of the form of equation \([6](https://arxiv.org/html/2610.41301#Sx2.E6)\); an answer formDDmoves by the same sum withkDk\_\{D\}\.

Before the validity floor and standardization, the expected reward over valid first lines of a policyπ\\pifor promptqqis

𝔼x∼π\(⋅∣q\)\[rJ\(x∣q\)\]=−KL\(π∥PJ\)−H\(π\)\+logZJ\(q\),PJ\(x∣q\)=exp⁡rJ​\(x∣q\)ZJ​\(q\),ZJ\(q\)=∑x∈𝒱⁡\(q\)exprJ\(x∣q\),\\begin\{gathered\}\\mathbb\{E\}\_\{x\\sim\\pi\(\\cdot\\mid q\)\}\\big\[r\_\{J\}\(x\\mid q\)\\big\]=\-\\mathrm\{KL\}\\big\(\\pi\\,\\\|\\,P\_\{J\}\\big\)\-H\(\\pi\)\+\\log Z\_\{J\}\(q\),\\\\ P\_\{J\}\(x\\mid q\)=\\frac\{\\exp r\_\{J\}\(x\\mid q\)\}\{Z\_\{J\}\(q\)\},\\qquad Z\_\{J\}\(q\)=\\sum\_\{x\\in\\mathcal\{V\}\(q\)\}\\exp r\_\{J\}\(x\\mid q\),\\end\{gathered\}\(9\)with𝒱⁡\(q\)\\mathcal\{V\}\(q\)the first lines the carrier rule admits,H⁡\(π\)H\(\\pi\)the entropy over them \(the pull toward typical completions\) andlog⁡ZJ​\(q\)\\log Z\_\{J\}\(q\)a prompt constant removed by standardization\. The fixed\-reference reward subtracts the start policy’sr0=log⁡π0​\(x∣q\)r\_\{0\}=\\log\\pi\_\{0\}\(x\\mid q\); withP0P\_\{0\},Z0Z\_\{0\}defined likewise,

𝔼x∼πt\(⋅∣q\)\[rJ−r0\]=−KL\(πt∥PJ\)\+KL\(πt∥P0\)\+log⁡ZJ​\(q\)−log⁡Z0​\(q\),\\begin\{split\}\\mathbb\{E\}\_\{x\\sim\\pi\_\{t\}\(\\cdot\\mid q\)\}\\big\[r\_\{J\}\-r\_\{0\}\\big\]&=\-\\mathrm\{KL\}\\big\(\\pi\_\{t\}\\,\\\|\\,P\_\{J\}\\big\)\+\\mathrm\{KL\}\\big\(\\pi\_\{t\}\\,\\\|\\,P\_\{0\}\\big\)\\\\ &\\quad\+\\log Z\_\{J\}\(q\)\-\\log Z\_\{0\}\(q\),\\end\{split\}\(10\)discounting completions the start policy already scores highly\. The reverse\-KL reward of on\-policy distillation[Gu et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib19);[Lu and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.41301#bib.bib21)subtracts the current student’s log\-probability instead \(expectation−KL\(πt∥PJ\)\+logZJ\(q\)\-\\mathrm\{KL\}\(\\pi\_\{t\}\\,\\\|\\,P\_\{J\}\)\+\\log Z\_\{J\}\(q\)\); the two agree at the first step and separate as the policy moves\.

### Emission and absorption

Both use the candidate pool \(starting\-policy completions, eight per prompt\)\. Withrz,kr\_\{z,k\}the score of instancek∈\{1,…,8\}k\\in\\\{1,\\dots,8\\\}of judgezzandsd¯q​\[⋅\]\\overline\{\\mathrm\{sd\}\}\_\{q\}\[\\cdot\]the within\-prompt standard deviation averaged over prompts, emission is

Λ=log⁡VdiffVsame,Vdiff=156​∑k≠lsd¯q​\[rT,k−rE,l\],Vsame=156​∑z∈\{T,E\}∑k<lsd¯q​\[rz,k−rz,l\],\\begin\{gathered\}\\Lambda=\\log\\frac\{V\_\{\\mathrm\{diff\}\}\}\{V\_\{\\mathrm\{same\}\}\},\\\\ V\_\{\\mathrm\{diff\}\}=\\frac\{1\}\{56\}\\sum\_\{k\\neq l\}\\overline\{\\mathrm\{sd\}\}\_\{q\}\\big\[r\_\{T,k\}\-r\_\{E,l\}\\big\],\\\\ V\_\{\\mathrm\{same\}\}=\\frac\{1\}\{56\}\\sum\_\{z\\in\\\{T,E\\\}\}\\,\\sum\_\{k<l\}\\overline\{\\mathrm\{sd\}\}\_\{q\}\\big\[r\_\{z,k\}\-r\_\{z,l\}\\big\],\\end\{gathered\}\(11\)the log ratio of between\-judge to same\-judge variation of the score difference, each over 56 pairings; pairings sharing an index \(same fine\-tuning seed, or same wording with the animal exchanged\) are excluded from both terms, as their shared idiosyncrasy would enter only the numerator\.

Absorption is the correlation of equation \([7](https://arxiv.org/html/2610.41301#Sx2.E7)\) over candidates,

absorption=corrc​\(d⁡\(c\),k⁡\(c\)\),\\mathrm\{absorption\}=\\mathrm\{corr\}\_\{c\}\\big\(d\(c\),\\,k\(c\)\\big\),\(12\)with advantages standardized within prompt \(equation \([5](https://arxiv.org/html/2610.41301#Sx2.E5)\)\); it has no scale, and the covariance movingSSiscorr⁡\(d,k\)​σd​σk\\mathrm\{corr\}\(d,k\)\\,\\sigma\_\{d\}\\,\\sigma\_\{k\}\. Two versions ofddshare pool, groups andkk: the training instances’ standardized difference \(the absorption of Table[1](https://arxiv.org/html/2610.41301#S0.T1); primary input of Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)\), and the within\-prompt centred score difference averaged over the pool’s stored instances \(eight; one for Qwen FT\), describing the judges\. Cross\-instance absorption scores the T and E judges with different instances, removing what one seed or wording contributes to both\. The student term is a central finite difference atθ0\\theta\_\{0\},

k⁡\(c\)≈log⁡πθ0\+ϵ​v​\(c\)−log⁡πθ0−ϵ​v​\(c\)2​ϵ,v=∇θB​\(θ0\),k\(c\)\\approx\\frac\{\\log\\pi\_\{\\theta\_\{0\}\+\\epsilon v\}\(c\)\-\\log\\pi\_\{\\theta\_\{0\}\-\\epsilon v\}\(c\)\}\{2\\epsilon\},\\qquad v=\\nabla\_\{\\theta\}B\(\\theta\_\{0\}\),\(13\)with the stepϵ\\epsilonselected per cell \(Supplementary Methods\)\.

Carrier\-feature decompositions projectddandkkwithin prompt onto 26 letter frequencies and the 100 commonest bigrams \(syllable\) or 14 number statistics \(number\), against within\-group permutation\. First\-token decompositions splitkkby least squares into the path through the first\-token \(attention\-sink\)[Xiao et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib39)state at an early layer, via the gradient of the candidate log\-likelihood with respect to that state, and the remainder\. Both are exact; the corresponding interventions are training conditions blocking the first\-token path or freezing the output column of the sink neuron, a massive\-activation neuron[Sun et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib14)\.

### Judge framings and matched judges

The SP sentence has three framings:*You love X*\(training\),*You hate X*\(inversion\) and*You have read about X*\(neutral mention\)\. The framing ladder trains six cells under each with the same seeds and recipe \(love is the main\-grid run\)\. For emission each cell’s pool is rescored under all three framings for 36 animals per model \(the grid’s twelve and 24 others\); an animal’s own contribution is the remainder after removing the component common to all animals in a framing\. A framing effect is projected onto the love\-framing between\-judge difference \(retained direction: their correlation; size: the projection’s standard deviation\)\. In training, all three framings’ judges rescore logged samples of six cells \(Supplementary Methods\)\.

Matched judges hold the words fixed\. SP judges name both animals in the same positions and exchange only the valence words, in eight paraphrases \(*You like sloths\. You dislike lemurs\.*against*You dislike sloths\. You like lemurs\.*\), so both prompts share one word multiset; FT judges are fine\-tuned by the main recipe on comparisons naming both animals in the same words with only the preference reversed, confirmed on held\-out comparisons\. Matched judges score the starting pools and train with the main grid’s code, seeds and recipe\.

### Reward interventions

Each intervention changes only the reward, on eight seed pairs of Llama · FT · syllable · sloth–lemur and Llama · SP · number · sloth–lemur \(rank values,α\\alphaladder\) or of the second alone \(per\-token mean, fixed reference\)\. Rank values replace a group’s eight scores by fixed values−3\.5\-3\.5to\+3\.5\+3\.5\(order kept exactly, spacing discarded\); the per\-token mean replaces the sum in equation \([4](https://arxiv.org/html/2610.41301#Sx2.E4)\) \(removing completion length\); the fixed reference subtracts the same first line’s log\-likelihood under the student’s base model without the persona sentence, the fixed start policy \(equation \([10](https://arxiv.org/html/2610.41301#Sx2.E10)\)\)\. Theα\\alphaladder keeps the shared reward, which permuting or reversing scores within a group would destroy, and with it format learning and almost all policy movement: it scores every training completion with both judges and trains on

r=c\+α​s,c=rown\+rother2,s=rown−rother2,r=c\+\\alpha s,\\qquad c=\\frac\{r\_\{\\mathrm\{own\}\}\+r\_\{\\mathrm\{other\}\}\}\{2\},\\qquad s=\\frac\{r\_\{\\mathrm\{own\}\}\-r\_\{\\mathrm\{other\}\}\}\{2\},\(14\)withrownr\_\{\\mathrm\{own\}\},rotherr\_\{\\mathrm\{other\}\}the scores \(equation \([4](https://arxiv.org/html/2610.41301#Sx2.E4)\)\) under the run’s own and the other judge, soc=r¯c=\\bar\{r\},s=±δ​rs=\\pm\\,\\delta r\(equation \([8](https://arxiv.org/html/2610.41301#Sx2.E8)\)\)\.α=1\\alpha=1reproduces the unmodified reward exactly,α=0\\alpha=0gives both runs the same reward,α=2\\alpha=2doubles the between\-judge difference\. Extended Data Table[4](https://arxiv.org/html/2610.41301#Sx5.T4)gives each condition’s policy movement \(KL on preference\-evaluation items\) and validity\.

### Behaviour

Answer evaluations recorded during training are each name’s log\-likelihood \(summed over surface variants\) at the first answer position of ten open questions, and the names’ log\-odds in 40 forced\-choice templates in both orders\. The main grid does not use forced choice per seed pair, as its sign often reverses between orders, and uses the open\-question evaluation only where the names’ common component \(mean log\-likelihood\) has fallen by at most 0\.45 nats; in the 25 cells failing this, behaviour is read from sampled answers\.

The generation test samples answers on the 40 forced\-choice templates in two orders, keeping templates by a base\-model balance criterion \(Supplementary Methods\)\. An answer is direct if its first word, lowercased and stripped of punctuation, begins with one of the names; anything else \(refusals, names inside a sentence such as*I prefer slugs*\) is an echo, reported separately as the control\. The tested quantity is half the difference between T\-trained and E\-trained students in the T name’s share of direct answers naming either animal \(form of equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\)\)\. The name\-form test measures the T\-minus\-E margin of four answer forms on the same templates \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)\)\.

### Trait pairs outside the grid

The across\-pair correlation of absorption with transfer adds SP configurations on animal pairs outside the grid, trained by the same recipe with their own candidate pools; transfer is the estimand of equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\)\. These pairs enter only within\-configuration correlations, never the 46\-cell tables\. Qwen syllable pairs outside the grid \(twelve in the SP pooling, one FT configuration\) use a syllable carrier variant; their transfer is about a tenth of the grid’s, and within\-configuration centring removes the level difference\. Pooled analyses use eight seed pairs per grid cell\.

### Dispositional traits

The axes are rude–polite, harsh–gentle and persuasive–neutral\. One side’s SP judge is the student’s own model with*You love being X\. Being X is what you value and you think about being X constantly\.*; the other replaces X\. Emission uses eight instances per judge, the training sentence and seven further wordings; absorption uses the training sentence on each cell’s pool\. Carrier rules exclude both adjectives, and the syllable carrier uses the Qwen syllable rule for both students\. The preference evaluation asks about the student’s own manner \(26 questions crossed with 24 templates, such as*What is your style when you reply?*and*I am X\.*; 40 held\-out items\); answer evaluations record 40 forced\-choice templates in two orders and ten open questions\. Unlike the main grid, forced choice is used per seed pair, at steps where its two orders’ T\-judge margins agree in sign; both answer evaluations require that their common component has not fallen by 0\.45 nats\.

Training follows the SP recipe \(80 steps\) on both students and carriers: twelve cells of 40 seed pairs \(Table[2](https://arxiv.org/html/2610.41301#S0.T2)\)\. For the effect sizedS=S¯/sd⁡\(S\)d\_\{S\}=\\bar\{S\}/\\mathrm\{sd\}\(S\)over seed pairs, the minimum detectable effect at 80% power is2\.8/n2\.8/\\sqrt\{n\}\(0\.44 at 40 pairs\)\. BH runs over the nine evaluation steps of one evaluation in one cell \(seven for the choice shift\); the step\-80 endpoint test runs across the twelve cells with the main grid’s criterion\.

### Judges that are not the student’s own model

Cross\-family judges \(same route and trait pair\) score the Llama student with Qwen3\-1\.7B and the Qwen student with Llama\-3\.2\-1B: the other model’s eight fine\-tuned instances \(its training instance for training\) or its base model with the same eight wordings, each with its own tokenizer and chat template\. Pools,kkand the preference evaluation are the student’s; same\-seed comparisons use eight seed pairs per cell\. Extended Data Table[6](https://arxiv.org/html/2610.41301#Sx5.T6)compares the judge conditions stage by stage; Supplementary Note[Supplementary Note 11: How much of the push the optimizer realizes, and where](https://arxiv.org/html/2610.41301#Sx16)summarizes the per\-cell checks of invalid share, end\-of\-training KL, completion length, judge reliability, the trait difference’s correlation with length and the open\-question common component\. The across\-cell relation of absorption toS80S\_\{80\}\(46 cells: 24 Llama, 22 Qwen\) is compared with the own\-model relation in the same cells\.

The rotation of the judge’s preference isρ=corr⁡\(dx,dg\)\\rho=\\mathrm\{corr\}\(d\_\{x\},d\_\{g\}\)between cross\-family and own\-model trait differences on the same candidates; withd⟂d\_\{\\perp\}the standardized part ofdxd\_\{x\}orthogonal todgd\_\{g\},

absorptionx=ρ​absorptiong\+\(1−ρ2\)1/2​corr​\(d⟂,k\)\\mathrm\{absorption\}\_\{x\}=\\rho\\;\\mathrm\{absorption\}\_\{g\}\+\\big\(1\-\\rho^\{2\}\\big\)^\{1/2\}\\,\\mathrm\{corr\}\\big\(d\_\{\\perp\},k\\big\)\(15\)separates inherited from new absorption; the decomposition is exact with the observedρ\\rho\. As an estimate of the rotation itself,ρ\\rhois also corrected for unreliability with the Spearman–Brown reliability[Spearman \(1910\)](https://arxiv.org/html/2610.41301#bib.bib49);[Brown \(1910\)](https://arxiv.org/html/2610.41301#bib.bib50)of the eight\-instance mean\. The emission decomposition crosses, within carrier and route, both students’ pools with both judge families over the six pairs \(44 combinations per route\) and partitions the variance ofΛ\\Lambdainto trait pair, pool, judge family, their interaction and a pair\-specific remainder, with the instance noise floor from the intervals ofΛ\\Lambda\.

The 8B judges are built on Llama\-3\.1\-8B\-Instruct[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib15)and Qwen3\-8B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib16)by the main recipe \(eight bf16 fine\-tuned seeds per animal, or the base model with the eight wordings\); they score every cell’s starting pool, with start\-policy stages as in the main grid, and train eight cells\. Trait alignment between two judge conditions is the correlation of their trait differences on the same candidates \(training instances\)\. A judge model’s own association with a candidate iskkcomputed with that model in place of the student\.

Under a cross\-family judge the first\-step direction isvv\(equation \([S1](https://arxiv.org/html/2610.41301#Sx21.E1)\)\) with that judge’s rewards on the SP\-route pool; recorded training completions are rescored as in Supplementary Methods\.

### Statistics

Transfer in a cell is tested by a sign\-flip test of the meanS80S\_\{80\}[Fisher \(1935\)](https://arxiv.org/html/2610.41301#bib.bib46)over all its seed pairs; pair sizes enter, so no fixed count of pairs in one direction is required \(with eight pairs the smallest one\-sidedppis1/2561/256\)\. Every cell has a base of eight seed pairs; those with lower expected power from start\-policy absorption, including the unmodified\-reward conditions of the trajectory and intervention experiments, have 16 to 28\. With about 2\.3 chance positives per direction expected across 46 cells at one\-sided 0\.05, a cell is established where the two\-sidedp<0\.05p<0\.05and the Benjamini–Hochberg[Benjamini and Hochberg \(1995\)](https://arxiv.org/html/2610.41301#bib.bib43)\(BH\)qqover the 46 is below 0\.10; the one\-sided count is given beside it\. Bootstrap intervals[Efron \(1979\)](https://arxiv.org/html/2610.41301#bib.bib48)over seed pairs agree with thettinterval and exclude zero in all eighteen established cells\.

The test is exact if a seed pair’s two students \(shared weights and prompt set, independent streams, different judges\) are exchangeable; as the streams are assigned, not randomized, exactness is conditional on substream exchangeability\. If a run’s margin change does not depend on its judge, swapping judge labels only flips the sign ofSS, and name asymmetries in token count or baseline likelihood cancel inSS\. One\-samplettand Wilcoxon signed\-rank tests[Wilcoxon \(1945\)](https://arxiv.org/html/2610.41301#bib.bib47)\(BH over the 46\) establish the same eighteen cells and none against the judge\. Since cells in a configuration share students, templates and seed numbers, Benjamini–Yekutieli control under arbitrary dependence[Benjamini and Yekutieli \(2001\)](https://arxiv.org/html/2610.41301#bib.bib44)is also applied: 10 cells remain belowq=0\.10q=0\.10\.

The further pairs of the 35 cells with more than eight are also tested alone \(Extended Data Table[2](https://arxiv.org/html/2610.41301#Sx5.T2); BH over the 35\)\. Fisher’s rule combines the two groups[Fisher \(1932\)](https://arxiv.org/html/2610.41301#bib.bib45)of pairs in each direction \(two\-sidedpptwice the smaller combined value, capped at one\)\. Cells without further pairs take one as the second value, so one rule, with BH, covers all 46\.

Tests fall into three tiers: the 46 transfer tests \(primary, corrected as above\); the reward interventions and matched\-judge training conditions \(secondary, each a paired sign\-flip test on the same seeds, uncorrected across them, with equal pairs per cell where a configuration is pooled\); and framings, letter substitution, trajectories, path and weight decompositions, name form and mention order \(descriptive: effect sizes, intervals, uncorrectedpp\)\. Behaviour from sampled answers is tested in the twelve sampled cells with BH across them \(smallestq=0\.28q=0\.28\)\. Resampling follows the design: seed pairs forSS; prompts for emission and absorption, as a prompt’s completions share its context \(these intervals are conditional on the eight judge instances\); cells for across\-cell correlations, withppfrom permutations within the eight configurations so that between\-configuration differences cannot produce them\.SSin such correlations is the Table[1](https://arxiv.org/html/2610.41301#S0.T1)value over all seed pairs; compared judge conditions use the same seeds\.

### Data availability

The evaluation items, judge wordings, carrier rules, the per\-cell and per\-seed\-pair results needed to reproduce all figures and tables, and the training and evaluation logs and candidate pools of the 46 cells are available at[https://github\.com/kkt94/subliminal\-rl](https://github.com/kkt94/subliminal-rl)\. Logged training completions and rewards, fine\-tuned judge weights and trained student checkpoints are available from the corresponding author on reasonable request\. The base models are publicly available\.

### Code availability

Code for training, judge construction, evaluation, analysis and the generation of all figures and tables is available at[https://github\.com/kkt94/subliminal\-rl](https://github.com/kkt94/subliminal-rl)under the MIT licence\.

## References

- I\. Aden\-Ali, N\. Golowich, A\. Liu, A\. Shetty, A\. Moitra, and N\. HaghtalabSubliminal effects in your data: a general mechanism via log\-linearity\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 517–543\.External Links:[Link](https://proceedings.mlr.press/v306/aden-ali26a.html),2602\.04863Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW),2306\.13649Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p11.1)\.
- Askinet al\.\(2026\)B\. Askin, M\. Ustaomeroglu, A\. Nayak, G\. Joshi, G\. Qu, and C\. Joe\-WongEmergent and subliminal misalignment through the lens of data\-mediated transfer\.InICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning,External Links:[Link](https://openreview.net/forum?id=1aaCfiMSwu),2605\.12798Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Baiet al\.\(2022\)Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon, C\. Chen, C\. Olsson, C\. Olah, D\. Hernandez, D\. Drain, D\. Ganguli, D\. Li, E\. Tran\-Johnson, E\. Perez, J\. Kerr, J\. Mueller, J\. Ladish, J\. Landau, K\. Ndousse, K\. Lukosuite, L\. Lovitt, M\. Sellitto, N\. Elhage, N\. Schiefer, N\. Mercado, N\. DasSarma, R\. Lasenby, R\. Larson, S\. Ringer, S\. Johnston, S\. Kravec, S\. El Showk, S\. Fort, T\. Lanham, T\. Telleen\-Lawton, T\. Conerly, T\. Henighan, T\. Hume, S\. R\. Bowman, Z\. Hatfield\-Dodds, B\. Mann, D\. Amodei, N\. Joseph, S\. McCandlish, T\. Brown, and J\. KaplanConstitutional AI: harmlessness from AI feedback\.arXiv preprint arXiv:2212\.08073\.External Links:2212\.08073,[Document](https://dx.doi.org/10.48550/arXiv.2212.08073),[Link](https://arxiv.org/abs/2212.08073)Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Benjamini and Hochberg \(1995\)Y\. Benjamini and Y\. HochbergControlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B \(Methodological\)57\(1\),pp\. 289–300\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1995.tb02031.x)Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p5.1),[Statistics](https://arxiv.org/html/2610.41301#Sx2.SSx13.p1.1)\.
- Benjamini and Yekutieli \(2001\)Y\. Benjamini and D\. YekutieliThe control of the false discovery rate in multiple testing under dependency\.The Annals of Statistics29\(4\),pp\. 1165–1188\.External Links:[Document](https://dx.doi.org/10.1214/aos/1013699998)Cited by:[Statistics](https://arxiv.org/html/2610.41301#Sx2.SSx13.p2.1)\.
- Betleyet al\.\(2026\)J\. Betley, N\. Warncke, A\. Sztyber\-Betley, D\. Tan, X\. Bao, M\. Soto, M\. Srivastava, N\. Labenz, and O\. EvansTraining large language models on narrow tasks can lead to broad misalignment\.Nature649\(8097\),pp\. 584–589\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09937-5)Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Blanket al\.\(2026\)C\. Blank, A\. Bhatia, S\. Rajamanoharan, A\. Conmy, and N\. NandaSubliminal learning is steering vector distillation\.InMechanistic Interpretability Workshop at the 43rd International Conference on Machine Learning \(ICML 2026\),External Links:[Link](https://openreview.net/forum?id=qlYv5y0cWf),2606\.00995Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p9.1)\.
- Brockerset al\.\(2026\)V\. C\. Brockers, R\. D\. Ventzke, V\. Neuhaus, B\. Hidalgo\-Ogalde, and V\. PriesemannLearning through noise: why subliminal learning works and when it fails\.arXiv preprint arXiv:2605\.23645\.External Links:2605\.23645,[Document](https://dx.doi.org/10.48550/arXiv.2605.23645),[Link](https://arxiv.org/abs/2605.23645)Cited by:[Transfer weakens](https://arxiv.org/html/2610.41301#S0.SSx9.SSS0.Px4.p2.1),[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p14.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p9.1)\.
- Brown \(1910\)W\. BrownSome experimental results in the correlation of mental abilities\.British Journal of Psychology3\(3\),pp\. 296–322\.External Links:[Document](https://dx.doi.org/10.1111/j.2044-8295.1910.tb00207.x)Cited by:[Judges that are not the student’s own model](https://arxiv.org/html/2610.41301#Sx2.SSx12.p2.2)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html),1706\.03741Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Cloudet al\.\(2026\)A\. Cloud, M\. Le, J\. Chua, J\. Betley, A\. Sztyber\-Betley, S\. Mindermann, J\. Hilton, S\. Marks, and O\. EvansLanguage models transmit behavioural traits through hidden signals in data\.Nature652\(8110\),pp\. 615–621\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10319-8)Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p4.1),[Transfer weakens](https://arxiv.org/html/2610.41301#S0.SSx9.SSS0.Px4.p2.1),[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Abstract](https://arxiv.org/html/2610.41301#abstract1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p14.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p6.1)\.
- Efron \(1979\)B\. EfronBootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176344552)Cited by:[Statistics](https://arxiv.org/html/2610.41301#Sx2.SSx13.p1.1)\.
- Fisher \(1932\)R\. A\. FisherStatistical methods for research workers\.4th edition,Biological Monographs and Manuals,Oliver and Boyd,Edinburgh\.Cited by:[Statistics](https://arxiv.org/html/2610.41301#Sx2.SSx13.p3.1)\.
- Fisher \(1935\)R\. A\. FisherThe design of experiments\.Oliver and Boyd,Edinburgh\.Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p5.1),[Statistics](https://arxiv.org/html/2610.41301#Sx2.SSx13.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:2407\.21783,[Document](https://dx.doi.org/10.48550/arXiv.2407.21783),[Link](https://arxiv.org/abs/2407.21783)Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p1.1),[Design and models](https://arxiv.org/html/2610.41301#Sx2.SSx1.p1.1),[Judges that are not the student’s own model](https://arxiv.org/html/2610.41301#Sx2.SSx12.p3.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ),2306\.08543Cited by:[First\-order account of one GRPO step](https://arxiv.org/html/2610.41301#Sx2.SSx5.p3.3),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p11.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Note:NIPS 2014 Deep Learning WorkshopExternal Links:1503\.02531,[Document](https://dx.doi.org/10.48550/arXiv.1503.02531),[Link](https://arxiv.org/abs/1503.02531)Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Huanget al\.\(2025\)A\. Huang, A\. Block, D\. J\. Foster, D\. Rohatgi, C\. Zhang, M\. Simchowitz, J\. T\. Ash, and A\. KrishnamurthySelf\-improvement in language models: the sharpening mechanism\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=WJaUkwci9o),2412\.01951Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p3.1),[Why the open\-question evaluation degenerates under same\-model judges](https://arxiv.org/html/2610.41301#S0.SSx7.SSS0.Px2.p1.1)\.
- Kim and Rush \(2016\)Y\. Kim and A\. M\. RushSequence\-level knowledge distillation\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,Austin, Texas,pp\. 1317–1327\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1139),[Link](https://aclanthology.org/D16-1139/)Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.In3rd International Conference on Learning Representations \(ICLR\),San Diego, CA, USA\.External Links:1412\.6980,[Link](https://arxiv.org/abs/1412.6980)Cited by:[One GRPO step and the four\-stage chain](https://arxiv.org/html/2610.41301#S0.SSx2.p2.1)\.
- Leeet al\.\(2024\)H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. R\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. PrakashRLAIF vs\. RLHF: scaling reinforcement learning from human feedback with AI feedback\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 26874–26901\.External Links:[Link](https://proceedings.mlr.press/v235/lee24t.html),2309\.00267Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding R1\-Zero\-like training: a critical perspective\.InSecond Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=5PAF7PAY2Y),2503\.20783Cited by:[Training](https://arxiv.org/html/2610.41301#Sx21.SSx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p10.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.In7th International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7),1711\.05101Cited by:[One GRPO step and the four\-stage chain](https://arxiv.org/html/2610.41301#S0.SSx2.p2.1),[Training](https://arxiv.org/html/2610.41301#Sx21.SSx1.p1.1)\.
- Lu and Thinking Machines Lab \(2025\)K\. Lu and Thinking Machines LabOn\-policy distillation\.Note:Thinking Machines Lab: Connectionism \(blog\)External Links:[Document](https://dx.doi.org/10.64434/tml.20251026),[Link](https://thinkingmachines.ai/blog/on-policy-distillation/)Cited by:[First\-order account of one GRPO step](https://arxiv.org/html/2610.41301#Sx2.SSx5.p3.3),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p11.1)\.
- Madl \(2026\)T\. MadlChannel location constrains the auditability of subliminal learning\.arXiv preprint arXiv:2606\.22019\.External Links:2606\.22019,[Document](https://dx.doi.org/10.48550/arXiv.2606.22019),[Link](https://arxiv.org/abs/2606.22019)Cited by:[Transfer weakens](https://arxiv.org/html/2610.41301#S0.SSx9.SSS0.Px4.p2.1),[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p14.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Magistraliet al\.\(2026\)I\. Magistrali, F\. Berdoz, S\. Dauncey, and R\. WattenhoferSubliminal signals in preference labels\.InAgents in the Wild: Safety, Security, and Beyond \(Workshop at the Fourteenth International Conference on Learning Representations, ICLR 2026\),External Links:[Link](https://openreview.net/forum?id=YnEIr2ZekL),2603\.01204Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p6.1)\.
- Morgulis and Hewitt \(2026\)G\. Morgulis and J\. HewittSubliminal steering: stronger encoding of hidden signals\.Note:arXiv preprint arXiv:2604\.25783External Links:2604\.25783,[Link](https://arxiv.org/abs/2604.25783)Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.External Links:[Document](https://dx.doi.org/10.52202/068431-2011),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html),2203\.02155Cited by:[Abstract](https://arxiv.org/html/2610.41301#abstract1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Panicksseryet al\.\(2024\)A\. Panickssery, S\. R\. Bowman, and S\. FengLLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 68772–68802\.External Links:[Document](https://dx.doi.org/10.52202/079017-2197),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html),2404\.13076Cited by:[Why the open\-question evaluation degenerates under same\-model judges](https://arxiv.org/html/2610.41301#S0.SSx7.SSS0.Px2.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 53728–53741\.External Links:[Document](https://dx.doi.org/10.52202/075280-2338),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html),2305\.18290Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p6.1)\.
- Schrodiet al\.\(2026\)S\. Schrodi, E\. Kempf, F\. Barez, and T\. BroxTowards understanding subliminal learning: when and how hidden biases transfer\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=IelhmYSjPt),2509\.23886Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Shahet al\.\(2026\)A\. Shah, J\. Chooi, J\. Ou, and S\. FengCovert influence between language models\.arXiv preprint arXiv:2606\.04071\.External Links:2606\.04071,[Document](https://dx.doi.org/10.48550/arXiv.2606.04071),[Link](https://arxiv.org/abs/2606.04071)Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p7.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300),[Link](https://arxiv.org/abs/2402.03300)Cited by:[Discussion](https://arxiv.org/html/2610.41301#Sx1.p1.1),[Design and models](https://arxiv.org/html/2610.41301#Sx2.SSx1.p1.1),[Abstract](https://arxiv.org/html/2610.41301#abstract1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p10.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Spearman \(1910\)C\. SpearmanCorrelation calculated from faulty data\.British Journal of Psychology3\(3\),pp\. 271–295\.External Links:[Document](https://dx.doi.org/10.1111/j.2044-8295.1910.tb00206.x)Cited by:[Judges that are not the student’s own model](https://arxiv.org/html/2610.41301#Sx2.SSx12.p2.2)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. ChristianoLearning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3008–3021\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html),2009\.01325Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p10.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Sunet al\.\(2024\)M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. LiuMassive activations in large language models\.InFirst Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=F7aAhfitX6),2402\.17762Cited by:[Absorption: the alignment shared across judge instances predicts transfer](https://arxiv.org/html/2610.41301#S0.SSx5.p4.1),[Emission and absorption](https://arxiv.org/html/2610.41301#Sx2.SSx6.p3.1),[Where absorption runs](https://arxiv.org/html/2610.41301#Sx21.SSx3.p1.1)\.
- von Werraet al\.\(2020\)L\. von Werra, Y\. Belkada, L\. Tunstall, E\. Beeching, T\. Thrush, N\. Lambert, S\. Huang, K\. Rasul, and Q\. GallouédecTRL: transformers reinforcement learning\.Note:GitHub repositoryExternal Links:[Link](https://github.com/huggingface/trl)Cited by:[Design and models](https://arxiv.org/html/2610.41301#Sx2.SSx1.p3.1),[Training](https://arxiv.org/html/2610.41301#Sx21.SSx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p10.1)\.
- Wanget al\.\(2026\)M\. Wang, Z\. Xu, J\. Fang, Y\. Yao, S\. Deng, H\. Chen, and N\. ZhangFrom data to behavior: predicting unintended model behaviors before training\.Note:arXiv preprint arXiv:2602\.04735Workshop on Principled Design for Trustworthy AI at ICLR 2026External Links:2602\.04735,[Link](https://arxiv.org/abs/2602.04735)Cited by:[The chain helps prevent part of this](https://arxiv.org/html/2610.41301#S0.SSx10.SSSx3.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511),[Link](https://aclanthology.org/2024.acl-long.511/)Cited by:[Which answer evaluation applies where](https://arxiv.org/html/2610.41301#S0.SSx7.SSS0.Px1.p1.1)\.
- Wataokaet al\.\(2024\)K\. Wataoka, T\. Takahashi, and R\. RiSelf\-preference bias in LLM\-as\-a\-judge\.arXiv preprint arXiv:2410\.21819\.Note:NeurIPS 2024 Safe Generative AI WorkshopExternal Links:2410\.21819,[Document](https://dx.doi.org/10.48550/arXiv.2410.21819),[Link](https://arxiv.org/abs/2410.21819)Cited by:[Why the open\-question evaluation degenerates under same\-model judges](https://arxiv.org/html/2610.41301#S0.SSx7.SSS0.Px2.p1.1)\.
- Wilcoxon \(1945\)F\. WilcoxonIndividual comparisons by ranking methods\.Biometrics Bulletin1\(6\),pp\. 80–83\.External Links:[Document](https://dx.doi.org/10.2307/3001968)Cited by:[Statistics](https://arxiv.org/html/2610.41301#Sx2.SSx13.p2.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF),2309\.17453Cited by:[Absorption: the alignment shared across judge instances predicts transfer](https://arxiv.org/html/2610.41301#S0.SSx5.p3.1),[Emission and absorption](https://arxiv.org/html/2610.41301#Sx2.SSx6.p3.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:2505\.09388,[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p1.1),[Design and models](https://arxiv.org/html/2610.41301#Sx2.SSx1.p1.1),[Judges that are not the student’s own model](https://arxiv.org/html/2610.41301#Sx2.SSx12.p3.1)\.
- Yuet al\.\(2025a\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 113222–113244\.External Links:[Document](https://dx.doi.org/10.52202/085713-3775),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html),2503\.14476Cited by:[Training](https://arxiv.org/html/2610.41301#Sx21.SSx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p10.1)\.
- Yuet al\.\(2025b\)T\. Yu, B\. Ji, S\. Wang, S\. Yao, Z\. Wang, G\. Cui, L\. Yuan, N\. Ding, Y\. Yao, Z\. Liu, M\. Sun, and T\. ChuaRLPR: extrapolating RLVR to general domains without verifiers\.arXiv preprint arXiv:2506\.18254\.External Links:2506\.18254,[Document](https://dx.doi.org/10.48550/arXiv.2506.18254),[Link](https://arxiv.org/abs/2506.18254)Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p11.1)\.
- Yuanet al\.\(2024\)W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. E\. WestonSelf\-rewarding language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 57905–57923\.External Links:[Link](https://proceedings.mlr.press/v235/yuan24d.html),2401\.10020Cited by:[A minimal experiment crossed into 48 cells](https://arxiv.org/html/2610.41301#S0.SSx1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.36,pp\. 46595–46623\.External Links:[Document](https://dx.doi.org/10.52202/075280-2020),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html),2306\.05685Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p8.1)\.
- Zhouet al\.\(2026\)X\. Zhou, Z\. Liu, A\. Sims, H\. Wang, T\. Pang, C\. Li, L\. Wang, M\. Lin, and C\. DuReinforcing general reasoning without verifiers\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=nnwvwge40d),2505\.21493Cited by:[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p11.1)\.
- Ziegleret al\.\(2019\)D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. IrvingFine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.External Links:1909\.08593,[Document](https://dx.doi.org/10.48550/arXiv.1909.08593),[Link](https://arxiv.org/abs/1909.08593)Cited by:[Training](https://arxiv.org/html/2610.41301#Sx21.SSx1.p1.1),[Subliminal Reinforcement Learning: traits travel through a judge’s scores](https://arxiv.org/html/2610.41301#p10.1)\.

## Acknowledgements

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant \(No\. RS\-2025\-25422680, No\. RS\-2020\-II201373\), and the National Research Foundation of Korea \(NRF\) grant \(No\. RS\-2025\-00520618\) funded by the Korean Government \(MSIT\)\.

## Author contributions

K\.K\. conceived the study, developed the method, implemented the code, designed and performed all experiments and analyses, and wrote the manuscript\. Y\.S\.C\. supervised the project and reviewed the manuscript\.

## Competing interests

The authors declare no competing interests\.

Extended Data Fig\. 1 \|:Every seed pair of every cell\.S80S\_\{80\}of each seed pair \(dots\), its range \(line\) and mean \(diamond\) for the 46 cells that pass the validity gate\. One seed pair \(Llama · FT · syllable · sloth–lemur,\+0\.279\+0\.279\) lies beyond the axis range of±0\.22\\pm 0\.22and is drawn at its limit\.Extended Data Fig\. 2 \|:The state of the optimization, and when transfer accrues\.a, Carrier validity by step\.b, Share of groups whose eight completions do not all receive the same reward\.c,SSover training in six cells\.Extended Data Table 1 \|:Each judge condition’s movement\.Change of the preference margin from step 0 to step 80 in the student trained under each judge, their mean \(the common push\) and half their difference \(SS\), over the eight base seed pairs\.Extended Data Table 2 \|:Transfer on a cell’s further seed pairs, and the combination of the two groups\.Extended Data Table 3 \|:Carrier channel decomposition, one row per cell\.Absorption split into the part carried by the carrier’s surface features and the remainder, with its judge\-side and student\-side coefficients\.Extended Data Table 4 \|:Reward intervention\.Training onc\+α​sc\+\\alpha s, whereccis the part of the reward shared by the two judges andssthe between\-judge part; eight seed pairs per condition\. Dashes mark theα=1\\alpha=1reference and values not logged for that run\.Extended Data Table 5 \|:The state of training, one row per cell\.Carrier validity, the share of groups that carry a reward signal, and when transfer accrues\.Extended Data Table 6 \|:Judges that are not the student’s own model\.For each route and student, the main\-grid judges beside the other family’s 1B\-class judges and 8B judges of either family, with everything else held fixed\.Supplementary Information

## Supplementary Note 1: The between\-judge contrast at the first step

In the between\-judge contrast of equation \([8](https://arxiv.org/html/2610.41301#Sx2.E8)\) the shared score cancels except in the rescaling term12​r¯~​\(σT,g−1−σE,g−1\)\\tfrac\{1\}\{2\}\\widetilde\{\\bar\{r\}\}\\,\(\\sigma\_\{T,g\}^\{\-1\}\-\\sigma\_\{E,g\}^\{\-1\}\), which vanishes when the two judges agree and is itself set by the trait\. On the start\-policy candidate pools,ddcorrelates with its trait part at 0\.87–0\.93 in six pools and at 0\.63–0\.66 in the two Qwen · FT · number pools measured\. Splitting the covariance withkkthe same way, the direct term carries the start\-policy alignment in six of the eight pools; the rescaling term offsets it by up to 45%, is the whole of the small alignment of Llama · SP · number · warbler–slug \(\+0\.033\+0\.033\) and nearly cancels the direct term in Qwen · FT · number · jerboa–badger\.

## Supplementary Note 2: The established set

Twenty\-four cells reach one\-sidedp<0\.05p<0\.05over all their seed pairs, 22 in the judge’s direction and two against it: Llama · SP · syllable · elephant–tiger \(16 seed pairs,−0\.041\-0\.041, 12 of 16 negative, one\-sidedp=0\.036p=0\.036\) and Llama · SP · number · elephant–tiger \(16 seed pairs,−0\.039\-0\.039, one\-sidedp=0\.028p=0\.028\)\. Under the main criterion these two drop \(two\-sidedp=0\.071p=0\.071,q=0\.149q=0\.149;p=0\.057p=0\.057,q=0\.137q=0\.137\), as do four cells in the judge’s direction, Llama · SP · syllable · sloth–lemur, Qwen · FT · syllable · otter–beaver, Qwen · FT · syllable · sloth–lemur and Qwen · SP · number · otter–beaver \(two\-sidedp=0\.063p=0\.063to 0\.082\)\. Seventeen of the eighteen established cells reachqqbelow 0\.05; the strongest are Qwen · SP · syllable · warbler–slug and Qwen · FT · number · warbler–slug, atq=0\.0007q=0\.0007\.

The established set is stable across tests \(Extended Data Table[2](https://arxiv.org/html/2610.41301#Sx5.T2)\)\. Fisher’s combination of each cell’s base eight seed pairs with its further pairs meets the criterion in 19 cells, ten of them atqqbelow 0\.05: 17 of the eighteen, together with Qwen · SP · number · shrew–cuttlefish in the judge’s direction and Llama · SP · syllable · elephant–tiger against it, in the direction its negative start\-policy absorption gives\. On their further pairs alone, 11 of the 35 cells with more than eight seed pairs meet the criterion, all in the judge’s direction and ten of them also established on all pairs\. The estimate on the further pairs is positive in 27 of the 35 \(two\-sided sign testp=0\.002p=0\.002, using no effect magnitude\) and averages\+0\.020\+0\.020with the 35 cells weighted equally\. With eight further pairs the smallest attainable two\-sidedppis 2/256, which five cells reach, so the smallestqqin that family is 0\.055\. One\-samplettand Wilcoxon tests establish sixteen cells atq<0\.05q<0\.05under either, against seventeen under the sign\-flip test\. Where sign patterns are drawn by Monte Carlo \(the six cells with 24 or 28 seed pairs\), nopplies closer than 0\.024 to the criterion\. Bootstrap intervals over seed pairs exclude zero in the eighteen established cells and also in two that are not established, Llama · SP · number · elephant–tiger and Qwen · SP · number · otter–beaver; the exact test is the more conservative of the two\.

Rescoring all twelve animal names in the trained students shows that the margin moves through one name of the judge’s pair\. In Qwen · SP · syllable · warbler–slug the T name \(warbler\) is the only one of the twelve that moves toward the T\-trained student \(\+0\.042\+0\.042, against−0\.011\-0\.011to−0\.043\-0\.043for the other eleven\) and exceeds the mean of the ten names outside the pair by\+0\.068\+0\.068\(7 of 8 seed pairs,p=0\.016p=0\.016\)\. In Qwen · FT · number · warbler–slug it is first of the twelve in all eight seed pairs \(\+0\.054\+0\.054above the mean of the other ten,p=0\.008p=0\.008\)\. In Qwen · SP · number · elephant–tiger the specific movement is on the other name: tiger falls below the mean of the ten by−0\.019\-0\.019in all eight seed pairs \(p=0\.008p=0\.008\), while elephant does not separate from them\.

Transfer is not monotone in the start\-policy stages\. Among cells without an established result, Llama · SP · number · shrew–cuttlefish holds the largest positive absorption in the grid \(\+0\.421\+0\.421\) and gives\+0\.007\+0\.007over 16 seed pairs \(one\-sidedp=0\.27p=0\.27\), whereas Qwen · FT · number · sloth–lemur, whose start\-policy emission is at the level of judge\-instance noise \(\+0\.004\+0\.004\[−0\.005\-0\.005,\+0\.013\+0\.013\]\), and Qwen · FT · number · jerboa–badger, with the fourth\-smallest emission in the grid \(\+0\.029\+0\.029\[\+0\.018\+0\.018,\+0\.042\+0\.042\]\), both transfer \(16 seed pairs each\)\.

Each established cell rests on at least 16 seed pairs\. Seventeen of the 28 cells without an established result have 16 to 28 seed pairs, not only eight\.

## Supplementary Note 3: Words implying an animal, and the concealment audit

The carrier rules block the names themselves\. The syllable rule lets through 0\.000–0\.016% of completions containing a name in Llama and none among the 38,000–40,500 scored completions per Qwen seed pair; in the syllable carrier 0\.06–0\.52% of all recorded completions contain a name as a substring, under the two judges alike \(0\.21% and 0\.16% of 327,680 each\), and the rule rejects almost all of them\.

Words that imply an animal are blocked by neither rule, and one cell uses them\. In Qwen · SP · syllable · warbler–slug, 13–18% of completions in the first 16 steps contain words implying the slug \(*squishy*,*squish*,*goo*,*gooey*,*slithery*\), more under the slug\-preferring judge \(16–18% against 13–15%\) and for longer \(2\.4–3\.2% against 0\.3–1\.5% in steps 16–47\), before they disappear from step 48; from then on the cell’s generations meet concealment as defined here, and its endpoint carries the earlier steps with it\. The same words are a habit of the Qwen syllable start policy regardless of the trait pair, so they are as frequent early in Qwen · FT · syllable · warbler–slug \(12\.6% against 12\.4%\), but there the two judges do not reward them differently and the cell is not established\.

The leak is the one place where three conditions meet, each measurable on the start\-policy candidates, in the judge\-term\-times\-student\-term structure of absorption at the level of word meaning\. The student must write the words often: 17\.1% of candidates contain a slug\-implying word, against at most 0\.3% for any other animal in either model\. The preference margin must be tied to them: within a prompt, moving the margin toward slug raises the probability of those candidates \(kklower by 0\.44 standard deviations, in 96% of prompts\), and warbler\-implying words are tied the other way \(\+0\.42\+0\.42\)\. And the two judges must score them differently: the slug persona favours slug\-implying candidates over the others in the same prompt by 0\.147 standard deviations ofdd, with the same sign under all eight wordings, whereas the warbler persona does not favour warbler\-implying words \(−0\.007\-0\.007to\+0\.025\+0\.025\) and the fine\-tuned judge barely separates slug\-implying ones \(−0\.025\-0\.025\)\. Removing those candidates lowers the cell’s start\-policy absorption only from\+0\.333\+0\.333to\+0\.308\+0\.308; they carry 26% of its covariance\. On the recorded training samples the share is similar, 15\.5% over the whole run \(20–24% in the first steps and none after step 48, when the late alignment of\+0\.38\+0\.38under the warbler judge is carried entirely by other candidates\), so most of the cell’s transfer runs through other channels\. Separately, Qwen syllable completions drift during training toward lists of real English words, mostly fruit names and number words, unrelated to the trait pairs\.

Without Qwen · SP · syllable · warbler–slug, with the false\-discovery rate recomputed over the remaining 45 cells, the other seventeen established cells remain established, sixteen of them atqqbelow 0\.05, with meanS80S\_\{80\}\+0\.032\+0\.032against\+0\.035\+0\.035for all eighteen; the largest cell becomes Llama · FT · syllable · sloth–lemur, whose completions carry no word implying an animal\. No stage of the chain rests on this cell\.

## Supplementary Note 4: Emission: carriers and per\-animal biases

Computed on the group\-standardized advantages with the same pairings,Λ\\Lambdacorrelates with the score\-based value at\+0\.997\+0\.997across cells, so standardization does not remove the trait signal\. The carrier sets most of its size: the spread of the score difference between judges given different animals exceeds that between instances given the same animal by about 40% in the syllable cells \(meanΛ\\Lambda\+0\.33\+0\.33\) and by about 10% in the number cells \(\+0\.10\+0\.10\), and the five smallest values are number cells\. The trait pair adds a consistent component beyond the configuration \(9% of the variance, within\-configuration permutationp<0\.0002p<0\.0002\): warbler–slug is above its configuration mean in 8 of 8 configurations, otter–beaver and elephant–tiger in 1 of 8\. Beyond its presence, emission does not order transfer: within configurations, cells above and below the median emission do not differ inS80S\_\{80\}\(p=0\.40p=0\.40\)\.

In the 22 syllable cells the letter coefficient of the judges’ score difference, its match with the letters of the judge’s own animal name, ranges from\+0\.023\+0\.023to\+0\.718\+0\.718, and that of the training instance’s standardized advantage from−0\.068\-0\.068to\+0\.671\+0\.671\(positive in 21; Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)\)\. In the substitution test, replacing a position that holds a letter absent from both names by a letter unique to the T name givesΔ​d\\Delta d\+0\.270\+0\.270andΔ​k\\Delta k\+19\.1\+19\.1in Llama · FT · syllable · otter–beaver, positive in 0\.69 and 0\.75 of 400 matched pairs \(tt\+8\.1\+8\.1and\+12\.4\+12\.4\)\.

In the number carrier, the persona ordering of the twelve animals follows how strongly each persona prefers larger numbers \(rr\+0\.72\+0\.72on the Llama candidate pool,\+0\.92\+0\.92on the Qwen pool\), is unrelated to the animals’ real sizes and differs between models \(rank correlation−0\.10\-0\.10between the two orders\)\. Fine\-tuned judges have no frame to rescore, so each student’s number pool is scored by the fine\-tuned judges of all twelve animals in all eight fine\-tuning seeds, with the student’s own weights as baseline\. The resulting per\-animal effect is far more stable than a persona’s: 98\.0% \(Qwen\) and 98\.8% \(Llama\) of its variance lies between animals rather than between seeds of the same animal, with split\-half reliability over seeds of 0\.996 and 0\.998\. Its ordering by association with number size is reproducible across seeds \(\+0\.80\+0\.80and\+0\.73\+0\.73\) but is not the persona ordering of the same student model \(rank correlation\+0\.14\+0\.14in Qwen\), so each route carries its own ordering\. Over the six trait pairs, the grid’sΛ\\Lambdafollows the spread of the difference between the two animals’ effect vectors,sd⁡\(e^T−e^E\)\\mathrm\{sd\}\(\\hat\{e\}\_\{T\}\-\\hat\{e\}\_\{E\}\), with rank correlation\+1\.00\+1\.00on the Qwen pool and\+0\.77\+0\.77on the Llama pool, and the trait\-pair component of the emission decomposition across judge families \(Methods\) follows it at\+0\.83\+0\.83and\+0\.71\+0\.71\. The dissimilarity measure that also serves for personas, the negative correlation of the two effect vectors, does not generalize: its rank correlation withΛ\\Lambdais\+0\.77\+0\.77on the Qwen pool and−0\.77\-0\.77on the Llama pool\.

Emission also forms on the samples the policy draws\. In the four Qwen · FT · number cells, in\-training emission does not separate them at the start \(elephant–tiger 0\.058 above jerboa–badger 0\.036, of which only jerboa–badger transfers\) but by the end ranks them in the order of their transfer \(warbler–slug, then jerboa–badger, then otter–beaver and elephant–tiger about equal\): jerboa–badger’s trait signal grows fourfold on the T run’s samples and twofold on the E run’s, while elephant–tiger’s shrinks\. In Qwen · FT · number · sloth–lemur, rescoring the training samples of eight seed pairs with all sixteen judges,Λ\\Lambdaon the T run’s samples is\+0\.018\+0\.018\[\+0\.013\+0\.013,\+0\.025\+0\.025\] over calls 0–16 and\+0\.050\+0\.050\[\+0\.031\+0\.031,\+0\.071\+0\.071\] over calls 56–79; on the E run’s samples it rises from\+0\.026\+0\.026to\+0\.049\+0\.049in mid\-training and stays positive \(\+0\.030\+0\.030\[\+0\.021\+0\.021,\+0\.041\+0\.041\] at the end\)\. For the training instance alone, with the same\-animal spread taken over its own pairings, it rises from 0\.000 on the start\-policy pool to\+0\.065\+0\.065\[\+0\.050\+0\.050,\+0\.081\+0\.081\] and\+0\.089\+0\.089\[\+0\.069\+0\.069,\+0\.109\+0\.109\] in mid\-training on the two runs’ samples\.

In two cells the judges’ trait difference is partly a per\-token likelihood offset of the judge pair \(correlation with completion length−0\.45\-0\.45in Llama · FT · syllable · jerboa–badger and\+0\.38\+0\.38in Qwen · FT · syllable · warbler–slug\); neither cell is established\.

On the group\-standardized advantagesΛ\\Lambdais positive in the same 45 cells as on the scores, and only in Qwen · FT · number · sloth–lemur does its interval contain zero\. In the letter\-substitution test the link between letters, the judges’ trait difference and the student’skkis causal: strongly over 400 matched pairs in Llama · FT · syllable · otter–beaver, with the same sign in Llama · SP · syllable · otter–beaver and Llama · SP · syllable · warbler–slug\. In the number carrier, twelve animal personas rescoring one pool in the same frame give each animal one number, its judge\-specific bias along the direction common to animals; a pair’s judge term, a covariance, is the difference of its animals’ numbers, so one ordering fixes its sign in all 66 pairings\.

## Supplementary Note 5: Judge framings and matched judges

Rescoring each pool with the judge’s frame changed leaves the persona size preferences largely in place\. Between*You love X*and*You hate X*, the twelve per\-animal values correlate atrr\+0\.86\+0\.86on the Llama pool \(signs agreeing for 10 of 12 animals\) and\+0\.75\+0\.75on the Qwen pool \(9 of 12\), and the pairwise emission direction correlates between the two frames at\+0\.72\+0\.72to\+0\.89\+0\.89across the six Llama pairs and\+0\.63\+0\.63to\+0\.82\+0\.82across the six Qwen pairs\. The neutral mention \(*You have read about X*\) keeps the per\-animal order in Qwen \(rr\+0\.75\+0\.75, 9 of 12\) and weakens it in Llama \(\+0\.40\+0\.40, 8 of 12\), while the pairwise direction stays between\+0\.50\+0\.50and\+0\.86\+0\.86across the Llama pairs and between\+0\.43\+0\.43and\+0\.61\+0\.61across the Qwen pairs\.

Over 36 animals per model, once the component common to all animals is removed, the retained direction of an animal’s own contribution averages\+0\.77\+0\.77under inversion and\+0\.73\+0\.73under the neutral mention in Llama, and\+0\.69\+0\.69and\+0\.44\+0\.44in Qwen, with the twelve grid animals and the other 24 in agreement \(\+0\.79\+0\.79against\+0\.76\+0\.76,\+0\.71\+0\.71against\+0\.74\+0\.74,\+0\.72\+0\.72against\+0\.67\+0\.67,\+0\.49\+0\.49against\+0\.42\+0\.42\)\. Survival under inversion follows the size of the animal’s own contribution relative to the common component—one property, since its size and its resemblance to the common component correlate at−0\.95\-0\.95to−0\.98\-0\.98—atrr\+0\.44\+0\.44overall and\+0\.49\+0\.49in each model separately; name token count adds a partial\+0\.28\+0\.28\. Under the neutral mention this holds in Qwen \(\+0\.46\+0\.46\) but not in Llama \(\+0\.08\+0\.08\), where survival is uniformly high \(\+0\.73\+0\.73, sd 0\.13\)\. At the level of the models, animal effects are 93–100% common component in both; Llama carries the common component and the idiosyncratic scale unchanged across framings, whereas Qwen’s neutral frame alters the common component \(rr\+0\.87\+0\.87\) and inflates the idiosyncratic part by 1\.28, the part that does not survive\.

In the training ladder \(six cells, same seeds, only the frame replaced\), love transfers in five; in Qwen · SP · number · shrew–cuttlefish it is not established\. Transfer is established under inversion in Llama · SP · number · warbler–slug, Qwen · SP · syllable · otter–beaver and Qwen · SP · syllable · warbler–slug, and under the neutral mention in Llama · SP · syllable · warbler–slug and Qwen · SP · syllable · warbler–slug, with Llama · SP · number · warbler–slug atp=0\.059p=0\.059\. A framing counts as lost where its transfer is significantly smaller than love’s in the paired comparison\. Inversion is lost alone in Llama · SP · syllable · warbler–slug, the best\-powered cell at 24 seed pairs \(\+0\.023\+0\.023love,\+0\.002\+0\.002inversion,\+0\.021\+0\.021neutral mention; inversion against love−0\.021\-0\.021,p=0\.036p=0\.036\), and together with the neutral mention in Llama · SP · number · sloth–lemur \(\+0\.089\+0\.089,\+0\.016\+0\.016,\+0\.026\+0\.026;−0\.073\-0\.073and−0\.063\-0\.063,p=0\.023p=0\.023and 0\.008\)\. The neutral mention alone is lost in Qwen · SP · syllable · otter–beaver \(\+0\.038\+0\.038,\+0\.053\+0\.053,\+0\.002\+0\.002; neutral mention against love−0\.037\-0\.037,p=0\.008p=0\.008\) and Qwen · SP · number · shrew–cuttlefish \(\+0\.021\+0\.021,\+0\.009\+0\.009,−0\.006\-0\.006;−0\.027\-0\.027,p=0\.039p=0\.039\)\. Neither is lost in Llama · SP · number · warbler–slug and Qwen · SP · syllable · warbler–slug\. Validity stays at or above 0\.909 throughout\. That an evaluative stance supplies the size therefore holds in the cells where inversion fails and is contradicted where the neutral mention fails instead, and one configuration can hold a trait pair of each kind\.

Which framing a cell needs is predicted by emission, not by absorption\. Splitting the reframed judges’ between\-judge difference into its component along the love\-frame difference and a residual, the retained direction orders the framings within these cells \(rr\+0\.55\+0\.55,p=0\.030p=0\.030\), and so does the size of that component \(\+0\.44\+0\.44,p=0\.035p=0\.035\), while absorption does not \(p=0\.11p=0\.11\)\. This resolves the two cases that look contradictory on absorption alone\. In Llama · SP · number · warbler–slug inversion has absorption−0\.042\-0\.042, the wrong sign, yet keeps direction 0\.860 and almost all of the size of the love\-direction component \(0\.031 to 0\.028\): the negative correlation comes from the residual, which is anti\-aligned withkk, and inversion transfers\. In Qwen · SP · syllable · otter–beaver the neutral mention has absorption\+0\.136\+0\.136against love’s\+0\.174\+0\.174, which looks equivalent, but its love\-direction component shrinks by 44% \(0\.083 to 0\.047\), with the lowest retained direction of any cell \(0\.768\), and the neutral mention fails\. Absorption, a correlation over the whole difference, mixes the retained component with the residual in the first case and discards the scale that separates the framings in the second\. The same measure holds off the start policy: on the logged training samples of six cells, rescored with the love judge, the retained direction predicts per\-framingSSwithin cells \(rr\+0\.59\+0\.59, within\-cell permutationp=0\.023p=0\.023, 17 framing–cell points\), whereas absorption on the same samples does not \(\+0\.50\+0\.50,p=0\.22p=0\.22\) and the first\-order GRPO push is unrelated \(\+0\.06\+0\.06,p=0\.77p=0\.77\)\. Robustness to reframing is fixed at emission: the retained direction measured on the judges’ scores matches that measured on absorption across five cells and two framings \(rr\+0\.72\+0\.72over ten points\), and inversion and neutral mention move together \(rr\+0\.97\+0\.97\), so robustness behaves as one number per cell\.

Matched persona judges, naming both animals in the same positions and exchanging only the valence words, emitΛ\\Lambda\+0\.019\+0\.019\[\+0\.017\+0\.017,\+0\.021\+0\.021\] in Llama · SP · number · sloth–lemur,\+0\.036\+0\.036\[\+0\.033\+0\.033,\+0\.039\+0\.039\] in Llama · SP · number · warbler–slug and\+0\.008\+0\.008\[\+0\.004\+0\.004,\+0\.013\+0\.013\] in Qwen · SP · number · elephant–tiger, against\+0\.103\+0\.103,\+0\.227\+0\.227and\+0\.124\+0\.124for the single\-animal wordings on the same candidates\. Across paraphrases the single\-animal between\-judge difference is stable \(split\-half correlation\+0\.81\+0\.81to\+0\.93\+0\.93\), whereas the matched one is largely wording\-specific \(\+0\.09\+0\.09to\+0\.54\+0\.54\), a third to a half of its size and unaligned with it \(−0\.22\-0\.22to\+0\.16\+0\.16across the three cells\); a control that changes only a pronoun \(*you always prefer the former*against*the latter*\) agrees no better \(−0\.16\-0\.16to\+0\.22\+0\.22\)\. On the start\-policy pool, then, the judges’ score difference is set mainly by which animal is named in the judge’s context, and the attitude expressed toward it sets its amplitude rather than its direction: an entity\-conditioned bias over carriers that cannot name the animal\. Trained end to end over 24 seed pairs each, the matched judges giveS80S\_\{80\}\+0\.027\+0\.027\[−0\.004\-0\.004,\+0\.058\+0\.058\] in Llama · SP · number · sloth–lemur and\+0\.008\+0\.008\[−0\.007\-0\.007,\+0\.023\+0\.023\] in Llama · SP · number · warbler–slug, pooled\+0\.018\+0\.018\[\+0\.000\+0\.000,\+0\.035\+0\.035\] \(29 of 48 positive, two\-sidedp=0\.057p=0\.057\); where the original wording was trained on the same seeds it gives\+0\.036\+0\.036against\+0\.058\+0\.058, a paired difference of−0\.022\-0\.022\(p=0\.12p=0\.12\)\. In the Qwen cell the matched judge barely moves the policy \(KL 0\.06 against 0\.8; estimate−0\.013\-0\.013\)\.

Judges fine\-tuned on comparisons naming both animals in the same words, in Llama · FT · syllable · otter–beaver, Llama · FT · syllable · shrew–cuttlefish and Llama · FT · syllable · warbler–slug, emitΛ\\Lambda\+0\.05\+0\.05to\+0\.08\+0\.08against\+0\.19\+0\.19to\+0\.21\+0\.21for the original judges, with retained direction−0\.07\-0\.07to\+0\.30\+0\.30and absorption−0\.13\-0\.13to−0\.01\-0\.01\. Trained on the same seeds, they move the policy \(KL 0\.84 to 1\.03\) and giveS80S\_\{80\}−0\.012\-0\.012,−0\.008\-0\.008and−0\.031\-0\.031\(pooled−0\.017\-0\.017over 24 seed pairs, two\-sidedp=0\.054p=0\.054\), against\+0\.022\+0\.022,\+0\.029\+0\.029and\+0\.018\+0\.018for the original judges on the same seed pairs \(per cell−0\.033\-0\.033\[−0\.054\-0\.054,−0\.011\-0\.011\],−0\.037\-0\.037\[−0\.063\-0\.063,−0\.014\-0\.014\] and−0\.049\-0\.049\[−0\.093\-0\.093,\+0\.003\+0\.003\]\)\.

Lexical association fixes the direction of the framed judges’ score difference; whether the evaluative stance supplies its size varies with the trait pair within a model and carrier\. The six reframed cells give the direction of the link between retained direction and per\-framing transfer, not its size\. In Llama · FT · syllable · otter–beaver, Llama · FT · syllable · shrew–cuttlefish and Llama · FT · syllable · warbler–slug, the single\-animal fine\-tuning data gave the judge the carrier ranking it transferred through; as the single\-animal and comparison fine\-tuning sets also differ in item structure, this identifies the construction, not lexical exposure alone\.

## Supplementary Note 6: Absorption: judge instances, trait pairs and paths

Across the 35 cells with eight judge instances on one pool, instance\-averaged absorption predictsSSin both routes \(FT\+0\.36\+0\.36, SP\+0\.54\+0\.54; togetherr=\+0\.49r=\+0\.49\)\. AsS80S\_\{80\}over eight seed pairs has reliability 0\.50–0\.60 against 0\.95 for the predictor, the observable correlation is capped near 0\.74\. Cross\-instance absorption, which scores the T and E judges with different instances and so removes what one wording or seed contributes to both, givesr=\+0\.48r=\+0\.48\(p=0\.011p=0\.011\)\. The excess of the training instance over the other seven is uninformative overall \(rr\+0\.08\+0\.08,p=0\.73p=0\.73\)\. In the 23 SP cells the excess tends to carry into transfer \(rr\+0\.53\+0\.53,p=0\.056p=0\.056\) and the training wording alone is the best single predictor \(rr\+0\.63\+0\.63,p=0\.009p=0\.009\); the SP relation holds when any one cell is dropped \(coefficient\+0\.38\+0\.38to\+0\.61\+0\.61\)\. Over the 23 FT cells the training seed does not predict transfer \(rr−0\.18\-0\.18,p=0\.42p=0\.42\)\. Over the 12 Llama FT cells it runs against transfer \(−0\.66\-0\.66,p=0\.017p=0\.017\), as does its excess over the other seven \(−0\.70\-0\.70,p=0\.012p=0\.012\), a relation confined to the two Llama FT configurations\. The raw alignment of the training instance does not survive dropping single trait pairs, one of which can reverse a configuration’s correlation, so the route difference is stated on the instance\-specific excess across cells\.

The pooled SP correlation over 40 trait pairs \(twelve of them in Qwen · SP · syllable\) is stable: dropping any one pair leaves it between\+0\.567\+0\.567and\+0\.778\+0\.778, dropping any one configuration between\+0\.612\+0\.612and\+0\.682\+0\.682, and the worst of all pairs of exclusions gives\+0\.528\+0\.528\. It has two separable parts\. The rank of the training wording among the pair’s eight instances predictsSSatrr\+0\.59\+0\.59\(p=0\.0013p=0\.0013;\+0\.53\+0\.53to\+0\.67\+0\.67leaving out any one pair\) and the instance mean at\+0\.50\+0\.50\(p=0\.0065p=0\.0065\), each surviving control for the other \(partialrr\+0\.51\+0\.51,p=0\.006p=0\.006, and\+0\.39\+0\.39,p=0\.038p=0\.038\); the spread across instances is unrelated \(−0\.24\-0\.24\)\. The rank part is a property of the pairs, not an effect of the wording: twelve pairs \(six Llama · SP · number, six Qwen · SP · number\) were trained on the same eight seeds under the pair’s most and least absorbing wording, and transfer is higher under the more absorbing wording in 3 of 12 pairs \(sign testp=0\.98p=0\.98; mean difference−0\.011\-0\.011\); within a pairSSdoes not follow absorption across the three wordings \(rr−0\.02\-0\.02over 36 conditions; within\-pair permutationp=0\.53p=0\.53\)\. Training statistics are the same across wordings \(KL 0\.89–1\.05, final reward−8\.5\-8\.5to−9\.7\-9\.7, completion length 24\.0–24\.1\), so the wording does not change the scale of the update either\.

On the FT route, 31 trait pairs in five configurations, each scored with all eight judge seeds and centred within configuration, giverr−0\.16\-0\.16\(p=0\.30p=0\.30\) for the training seed,\+0\.03\+0\.03\(p=0\.83p=0\.83\) for the instance mean,\+0\.07\+0\.07cross\-instance and\+0\.05\+0\.05for the judge average\. The Qwen FT configuration on the syllable carrier variant is flat on its own \(\+0\.01\+0\.01and−0\.03\-0\.03\), and the negative coefficient of the two Llama FT configurations \(−0\.66\-0\.66over 12 pairs\) disappears once the other configurations are included\. A fine\-tuned instance’s direction agrees with the instance mean at a median 0\.602, and 80% of what it measures is seed\-specific; persona wordings agree with their mean at a median 0\.945 in Llama and 0\.940 in Qwen, with an instance\-specific share of 0\.33 \(permutationp<0\.0001p<0\.0001against FT\)\.

On the student side, about 80% of absorption arises in layers 0–3, more in the MLPs than in attention\. The Llama sink neuron \(layer 1, neuron 1417\) has a first\-token activation2\.1×1052\.1\\times 10^\{5\}times the median; the Qwen one \(layer 2, neuron 1821\) is5\.3×1055\.3\\times 10^\{5\}times the median and carries 67–71% of the gradient energy of the preference margin in that layer\. Controlling base likelihood, length and per\-token likelihood moves the five negative cells only from−0\.294\-0\.294to−0\.301\-0\.301on average\.

In the six Llama · SP · number pairs the judge term is the difference of two per\-animal numbers, and its product with the student term gives the sign of absorption in 6 of 6; the same sign holds when the carrier is changed in the four pairs whose absorption is significant\. In Qwen · SP · number the rule holds in 3 of 3, the near\-zero absorption of elephant–tiger being a cancellation between the path along the common direction \(\+0\.195\+0\.195\) and the orthogonal path \(−0\.154\-0\.154\)\. Over all 66 persona pairs the product gives the sign in 48, and in 33 of the 40 with\|absorption\|≥0\.1\|\\mathrm\{absorption\}\|\\geq 0\.1\. The gradient of each name at the first\-token state makes a nearly uniform angle with the direction common to animal names \(cosine 0\.71–0\.86 in Llama, with warbler the one outlier at 0\.48; 0\.77–0\.91 in Qwen\), so names differ in gradient size: the correlation of the projectiong⋅μ^g\\cdot\\hat\{\\mu\}with\|g\|\|g\|is\+0\.66\+0\.66in Llama and\+0\.96\+0\.96in Qwen, and the student term equalssign⁡\(\|gT\|−\|gE\|\)\\mathrm\{sign\}\(\|g\_\{T\}\|\-\|g\_\{E\}\|\)in 12 of 12 pairs\. Gradient size is larger for the name with lower base probability in the evaluation context \(correlation of\|g\|\|g\|with that log\-likelihood−0\.72\-0\.72in Llama,−0\.83\-0\.83in Qwen\)\. Two pairs deviate from this probability rule \(Llama jerboa–badger, Qwen elephant–tiger\); in both the gradient size is not what the base probability alone predicts\.

In the carrier decomposition \(Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)\), letter\-composition features carry−0\.035\-0\.035to\+0\.166\+0\.166of the alignment across the 22 syllable cells\. In the five Qwen · SP · syllable cells the residual is the name\-specific component of the first\-token gradient, orthogonal to the common direction, which accounts for\+0\.069\+0\.069to\+0\.101\+0\.101of alignment while the component outside the first\-token path stays within±0\.013\\pm 0\.013\. Number features carry the alignment where the model has a strong size preference \(\+0\.191\+0\.191in Qwen · FT · number · warbler–slug,\+0\.093\+0\.093in Qwen · SP · number · jerboa–badger,−0\.070\-0\.070in Qwen · SP · number · sloth–lemur\) and little in the twelve Llama number cells \(\|part\|≤0\.056\|\\text\{part\}\|\\leq 0\.056\), except in Llama · SP · number · warbler–slug, whose small alignment \(\+0\.033\+0\.033\) is matched by its feature part \(\+0\.043\+0\.043\)\. The same decomposition of the judges’ score difference averaged over instances gives feature parts that correlate with these at\+0\.92\+0\.92across cells, with the same sign in 44 of 46\. Standardization itself changes little: over the 35 cells with eight instances, absorption from the averaged score difference and from the averaged standardized difference correlate at\+0\.989\+0\.989, with the same sign in 35 of 35\. In a cell whose alignment is small the training instance can still differ from both, as Llama · SP · number · warbler–slug shows \(\+0\.033\+0\.033against−0\.031\-0\.031and−0\.010\-0\.010\)\. The instance\-averaged score difference and the training instance’s absorption \(Table[1](https://arxiv.org/html/2610.41301#S0.T1)\) differ in sign in four cells, three of them because one instance trains rather than the mean, and only Qwen · FT · number · otter–beaver, whose pool stores one instance, because of standardization \(−0\.017\-0\.017against\+0\.051\+0\.051\); only in Llama · FT · number · jerboa–badger does the training instance’s interval exclude zero\. Extended Data Table[3](https://arxiv.org/html/2610.41301#Sx5.T3)takeskkat one step size for every cell, where Table[1](https://arxiv.org/html/2610.41301#S0.T1)uses the step selected per cell, so the two absorptions can differ in the third decimal \(at most 0\.007\)\.

In Llama · SP · syllable · jerboa–badger and Llama · SP · syllable · elephant–tiger, the component outside the first\-token path, tied directly to the candidate token positions, is−0\.070\-0\.070and−0\.111\-0\.111against a first\-token path of−0\.001\-0\.001and\+0\.135\+0\.135\. In the negative Llama · SP · number cells the first\-token path carries−0\.361\-0\.361,−0\.296\-0\.296and−0\.349\-0\.349of−0\.416\-0\.416,−0\.349\-0\.349and−0\.466\-0\.466in elephant–tiger, jerboa–badger and otter–beaver, and absorption is positive under 1, 0 and 1 of the eight wordings, against 8 of 8 for shrew–cuttlefish\. In the Qwen · SP · syllable comparison cell the component outside the first\-token path is 2% and the effect is absent\. The interventions that leave transfer in place comprise 96 runs blocking the first\-token path in Llama · FT · syllable, 80 in Qwen · SP · syllable and 80 freezing the sink neuron’s output column; the Llama effect over the six blocked cells stays at\+0\.026\+0\.026to\+0\.044\+0\.044\.

Within configurations absorption relates moderately to emission \(r=\+0\.32r=\+0\.32, within\-configuration permutationp=0\.022p=0\.022\), which accounts for about a tenth of its variance \(Fig\.[3](https://arxiv.org/html/2610.41301#S0.F3)a\)\. The routes differ in why the training instance’s own alignment does or does not carry into transfer\. A persona wording is given at every step, so its idiosyncrasy persists to step 80, and in SP cells the training wording is the best single predictor\. A fine\-tuned instance is a separate reward model whose seed\-specific preference on a fixed pool is not carried through the shift in sampling distribution during training, and in FT cells the training seed does not predict transfer\.

Controlling base likelihood, length and per\-token likelihood leaves the sign of absorption unchanged in all 46 cells, so the first\-token channel is not a formatting artefact\. In the carrier decomposition the syllable residual is positive in 19 of 22 cells\. In Llama · SP · syllable · jerboa–badger and Llama · SP · syllable · elephant–tiger, the only syllable cells with negative or near\-zero absorption despite a positive letter channel, the student gradient outside the first\-token path, tied directly to the candidate token positions, carries 9–19% of the gradient variance and opposes the persona’s preference; in the negative Llama · SP · number cells a negative student term on the first\-token path carries most of the negative absorption, with the same sign across persona wordings\.

Under first\-token blocking or freezing of the sink neuron’s output column, Qwen · SP · syllable · elephant–tiger, Qwen · SP · syllable · otter–beaver and Qwen · SP · syllable · warbler–slug remain established, and KL, reward and completion length are unchanged\.

## Supplementary Note 7: Transfer on the moving policy

In Llama · SP · number · warbler–slug, whose start\-policy absorption is\+0\.033\+0\.033\[−0\.010\-0\.010,\+0\.076\+0\.076\], the T run’s in\-training alignment grows from\+0\.003\+0\.003to\+0\.057\+0\.057while the E run’s falls from\+0\.026\+0\.026to−0\.047\-0\.047, and the student’s first\-token state carries the change \(R2R^\{2\}0\.97–0\.98 between the stored gradient and the first\-token path\)\. In Qwen · FT · number · sloth–lemur the training judge’s alignment runs from−0\.10\-0\.10/−0\.18\-0\.18early to\+0\.06\+0\.06,\+0\.24\+0\.24and\+0\.28\+0\.28at steps 64, 72 and 79 in the T run, and the first\-order trait share of each seed pair correlates with itsSSat\+0\.99\+0\.99over the four recorded pairs\. In Qwen · FT · syllable · elephant–tiger the T run’s alignment reverses from\+0\.26\+0\.26to−0\.38\-0\.38within the run in four logged seed pairs, all of which transfer \(\+0\.045\+0\.045\)\. The two fine\-tuned syllable elephant–tiger cells reverse by different routes: in Llama the letter channel’s explanatory power vanishes late \(R2R^\{2\}0\.01\) and the residual flips, while in Qwen the letter share itself goes from\+0\.065\+0\.065to−0\.135\-0\.135\. In Llama · SP · syllable · elephant–tiger the component outside the letter channel is negative at every phase under the training wording \(−0\.047\-0\.047,−0\.101\-0\.101and−0\.024\-0\.024early, middle and late\), most negative in mid\-training, where seed\-pair outcomes form; its near\-zero start\-policy absorption reflects averaging over the eight persona wordings, which inflates the letter share, and the wording used in training never aligns\.

Three mechanisms move the alignment\. Region drift: in Qwen · FT · number · elephant–tiger the shared reward moves the policy toward small numbers \(mean value 9\.5 to 3–6\), where the fixed relation betweenddandkkis negative\. Selective collapse: in Qwen · SP · number · shrew–cuttlefish the alignment grows from\+0\.09\+0\.09to\+0\.26\+0\.26as prompts with positive alignment keep their diversity while the number of effective prompts falls from 25 to 13–18, which accounts for 36–64% of the growth\. Re\-draw within prompts: in the syllable cells it matches the observed reversal, while region drift predicts the opposite sign\. In the Qwen number cells the change is carried inside the subspace of number statistics, whose share of the variance ofddandkkrises from 0\.1–0\.3 early to 0\.4–0\.9 late\.

Across the 48 judge–trait\-pair combinations of Llama · FT · number, judges with positive start\-policy absorption give a meanS80S\_\{80\}of\+0\.0109\+0\.0109\(29 combinations\) and judges with zero or negative absorption−0\.0134\-0\.0134\(19\)\. In Llama · FT · syllable absorption is positive for 40 of 42 combinations, leaving no range to test the relation, and transfer is positive in 40 of 42\. A fine\-tuned judge strongly prefers candidates along the common direction of the student’s first\-token state \(\+0\.67\+0\.67\), but almost identically for T and E \(99\.6% shared\), so its surviving judge term is small and changes with the fine\-tuning seed \(−0\.18\-0\.18and\+0\.05\+0\.05in two pairs\)\. For elephant–tiger, with the same student term \(−0\.090\-0\.090\), the persona judge’s term is\+0\.41\+0\.41: the difference between the routes sits in the judge term\.

Three Llama · SP · number pairs were trained under the wordings of highest and lowest start\-policy absorption; two have 16 seed pairs under the training wording, eight of them also trained under the two other wordings\. In elephant–tiger the negative transfer holds in both halves of the seed pairs under the training wording \(−0\.043\-0\.043and−0\.035\-0\.035\) but not under the other wordings on the same seeds \(\+0\.002\+0\.002and\+0\.013\+0\.013; paired differences\+0\.044\+0\.044,p=0\.016p=0\.016, and\+0\.056\+0\.056,p=0\.039p=0\.039\)\. In sloth–lemur the halves give\+0\.089\+0\.089and\+0\.031\+0\.031under the training wording and the other wordings\+0\.023\+0\.023and\+0\.036\+0\.036, within that range, so seed variation accounts for the difference\. Across the nine conditions no start\-policy quantity ordersSS\(within\-pair centring, nine points, none established\): absorption\+0\.01\+0\.01, the size of the judges’ trait difference\+0\.27\+0\.27, their covariance−0\.07\-0\.07, the share of the common component along the preference margin\+0\.15\+0\.15, the ratio of the two judges’ reward spreads−0\.04\-0\.04and the number\-size slope of the judge’s preference−0\.19\-0\.19\. The runs are otherwise indistinguishable \(final reward−8\.5\-8\.5to−9\.7\-9\.7, entropy 0\.080–0\.100, completion length 24\.0–24\.1\)\.

Under fixed ranks, transfer is\+0\.060\+0\.060against\+0\.073\+0\.073in Llama · FT · syllable · sloth–lemur and\+0\.095\+0\.095against\+0\.089\+0\.089in Llama · SP · number · sloth–lemur \(paired differences−0\.013\-0\.013,p=0\.73p=0\.73, and\+0\.006\+0\.006,p=0\.78p=0\.78\), with 8 of 8 seed pairs in the judge’s direction in both\. Training stays in the same regime \(validity 0\.817 against 0\.820, end reward−32\.1\-32\.1against−30\.8\-30\.8, completion length 63\.0 against 64\.1\), with the policy travelling somewhat less far \(KL 0\.75 against 1\.13\)\. In Llama · SP · number · sloth–lemur, the summed reward pushes the student toward short, typical completions \(29\.5 characters at step 80\) and moves the policy far \(KL 0\.91\); the per\-token mean keeps completions near the start policy’s length \(79\.2 characters\) and moves the policy a sixth as far \(KL 0\.14\)\. Transfer survives it:\+0\.033\+0\.033, six of eight seed pairs in the judge’s direction \(p=0\.039p=0\.039\), against\+0\.089\+0\.089for the summed reward \(paired difference−0\.055\-0\.055,p=0\.031p=0\.031\); per unit of policy movement rather more of it arrives\. Under the fixed\-reference reward \(equation \([10](https://arxiv.org/html/2610.41301#Sx2.E10)\)\) transfer is\+0\.043\+0\.043on the same seed pairs, eight of eight in the judge’s direction \(two\-sidedp=0\.0078p=0\.0078\), against\+0\.089\+0\.089\(paired difference−0\.046\-0\.046,p=0\.16p=0\.16\), with the policy again moving about a sixth as far \(KL 0\.16\)\.

In the reward intervention \(Extended Data Table[4](https://arxiv.org/html/2610.41301#Sx5.T4)\),α=0\\alpha=0in Llama · FT · syllable · sloth–lemur gives−0\.034\-0\.034with two of eight seed pairs in the judge’s direction \(p=0\.24p=0\.24\), the null expected when both runs receive the same reward and draw their samples independently, andα=2\\alpha=2gives\+0\.118\+0\.118with eight of eight \(p=0\.008p=0\.008\)\. End\-of\-training reward is matched acrossα=0\\alpha=0, 1 and 2 \(−30\.4\-30\.4,−30\.8\-30\.8and−31\.9\-31\.9\)\. In Llama · SP · number · sloth–lemur,α=0\\alpha=0gives five of eight seed pairs in the judge’s direction \(p=0\.89p=0\.89\), with training again matched \(end reward−9\.4\-9\.4\)\.

Under fine\-tuned syllable judges, elephant–tiger reverses out of the judge’s direction in both students, and neither cell is established; the reversal does not stop transfer in the logged seed pairs of Qwen · FT · syllable · elephant–tiger\.

In Qwen · FT · number · elephant–tiger, where the shared reward drifts the policy towards small numbers and the judges’ score difference relates negatively to the student’s response, the alignment flips from\+0\.31\+0\.31to−0\.10\-0\.10; reweighting early samples to match late ones reproduces 61–77% of the flip\. Judge and student gradient are fixed per candidate, so alignment moves only with which candidates compete in which prompt\.

Across the 48 judge–trait\-pair combinations of Llama · FT · number, centred within trait pair, the permutation test of the correlation between start\-policy absorption and transfer givesp=0\.0001p=0\.0001\. In elephant–tiger, a judge instance with absorption−0\.41\-0\.41givesS80=−0\.093S\_\{80\}=\-0\.093and one with\+0\.42\+0\.42gives\+0\.092\+0\.092\. Averaged over instances, absorption and transfer sit near zero, and the configuration has no established cell\.

Start\-policy absorption of Llama · SP · number · elephant–tiger is negative under all eight wordings, so negative alignment does not suffice for negative transfer\. The final KL of the runs under the three wordings is 0\.89–1\.05, and an independent preference evaluation on different items agrees with the main one\.

Permuting or reversing the scores within a group, the alternative to thec\+α​sc\+\\alpha ssplit, also destroys the shared part, and with it format learning and almost all policy movement \(KL 0\.001 against 1\.13\); thec\+α​sc\+\\alpha ssplit keeps both\. Doublingss\(α=2\\alpha=2\) enlarges transfer in Llama · FT · syllable · sloth–lemur \(\+0\.118\+0\.118against\+0\.073\+0\.073,p=0\.24p=0\.24\) and not in Llama · SP · number · sloth–lemur \(\+0\.045\+0\.045against\+0\.089\+0\.089,p=0\.13p=0\.13\)\.

## Supplementary Note 8: From the preference margin to the answer

The preference evaluation records each name’s log\-likelihood in two parts, the first name token and the remaining tokens, which locates where a shift is carried\. The forced\-choice evaluation cannot be used per seed pair: besides its sign reversing between presentation orders, in 25 cells the common component of the two names drops by 0\.45 to 5\.9 nats\.

The degeneration of the open\-question evaluation was traced in Llama · FT · syllable · sloth–lemur, Llama · FT · syllable · shrew–cuttlefish, Llama · SP · syllable · sloth–lemur and Llama · SP · syllable · warbler–slug, each under its own and under cross\-family judges on the same pool, through the gradient of the common componentCCand the push of equation \([S2](https://arxiv.org/html/2610.41301#Sx21.E2)\)\. An animal name as the first word of an answer has a start\-policy log\-likelihood of−16\-16to−23\-23nats\. On the FT route, the same\-model judge’s shared score follows the student’s base likelihood atr=0\.98r=0\.98\(cross\-family 0\.95\), its correlation withkCk\_\{C\}is−0\.124\-0\.124and−0\.112\-0\.112\(cross\-family−0\.099\-0\.099and−0\.103\-0\.103\), and its first\-order push onCCis 1\.28 and 1\.12 times the cross\-family judge’s\. Both conditions lowerCCby the end of the format phase \(−0\.96\-0\.96and−0\.78\-0\.78at step 24\); the difference arises afterwards\. The same\-model policy keeps sharpening between steps 50 and 80 \(entropy 0\.124 to 0\.072\), while the cross\-family policy stops at 0\.083 with more groups that no longer carry a signal \(0\.36 against 0\.30\), and at step 80CCis−0\.74\-0\.74against\+0\.05\+0\.05\. Within each condition, cells that sharpened more have lowerCC\(rr\+0\.59\+0\.59and\+0\.65\+0\.65with final entropy\): a same\-model judge’s score is almost the student’s own likelihood, so sharpening keeps raising it\. On the SP route the shared score equals the student’s base likelihood \(r=1\.00r=1\.00; cross\-family 0\.90\), the anti\-alignment is stronger \(−0\.238\-0\.238and−0\.293\-0\.293; cross\-family−0\.148\-0\.148and−0\.209\-0\.209\), the same\-model push onCCis 1\.66 and 1\.42 times the cross\-family one, and the drop at step 24 is correspondingly deeper \(−3\.16\-3\.16against−1\.15\-1\.15\)\. Both conditions recover in part, to−0\.61\-0\.61and−0\.13\-0\.13at step 80; on this route the depth of the early drop, rather than late sharpening, accounts for most of the gap, and associations within a condition are weak \(\|r\|≤0\.33\|r\|\\leq 0\.33\)\.

The two changes in sampled choice are Llama · SP · number · warbler–slug \(\+0\.0047\+0\.0047, 7 of 8 seed pairs, one\-sidedp=0\.012p=0\.012\) and Qwen · SP · number · shrew–cuttlefish \(\+0\.0086\+0\.0086, 6 of 8,p=0\.035p=0\.035\), two\-sidedp=0\.024p=0\.024and 0\.070\. The second falls just short of the transfer criterion \(20 seed pairs, one\-sidedp=0\.054p=0\.054\), and its route to the choice runs through the forced\-choice templates: in Qwen cells the choice predicted from template log\-odds matches the sampled choice \(rr\+0\.59\+0\.59to\+0\.84\+0\.84\), and here a log\-odds shift of\+0\.099\+0\.099on templates whose choice is still sensitive predicts a choice change of\+0\.012\+0\.012\. Qwen · SP · number · otter–beaver has a larger log\-odds shift \(\+0\.121\+0\.121,p=0\.012p=0\.012\), but on templates whose choice is already determined, so only the order of mention moves \(\+0\.0084\+0\.0084, 7 of 8,p=0\.027p=0\.027\); Llama · SP · number · elephant–tiger shifts the order of mention by−0\.0177\-0\.0177\(1 of 8,p=0\.012p=0\.012\) while its direct\-answer share does not move\. In the Llama syllable cells with a positive transfer estimate the direct\-answer choice moves by\+0\.0009\+0\.0009,\+0\.0007\+0\.0007,\+0\.0017\+0\.0017\(5 of 8,p=0\.25p=0\.25\) and\+0\.0015\+0\.0015in Llama · FT · syllable · otter–beaver, Llama · FT · syllable · shrew–cuttlefish, Llama · FT · syllable · warbler–slug and Llama · FT · syllable · sloth–lemur, and by\+0\.0007\+0\.0007in Llama · SP · syllable · sloth–lemur\. Llama · SP · syllable · shrew–cuttlefish moves it the other way \(−0\.0064\-0\.0064, 2 of 8,p=0\.023p=0\.023\) although its transfer is positive; its form margins locate the cause: the movement is in the lowercase form alone \(−0\.188\-0\.188, 1 of 8,p=0\.016p=0\.016\) while the capitalized form does not move \(−0\.007\-0\.007\), and the base answers under the two judges use the lowercase form at very different rates \(488 against 220\)\. In Llama · SP · syllable · sloth–lemur only the form “The sloth\.” spreads \(\+0\.116\+0\.116, 7 of 8,p=0\.008p=0\.008\), and in Llama · FT · syllable · shrew–cuttlefish the forms cancel\. Table[1](https://arxiv.org/html/2610.41301#S0.T1)gives the answer\-level result of every cell\.

In the per\-form margins, Llama · SP · number · sloth–lemur has\+0\.089\+0\.089of transfer against\+0\.027\+0\.027for Llama · SP · number · warbler–slug\. In warbler–slug the transfer reaches the capitalized singular \(\+0\.181\+0\.181, 7 of 8,p=0\.012p=0\.012\) while the plural falls \(−0\.171\-0\.171\); in sloth–lemur the capitalized singular does not move \(−0\.031\-0\.031\) and the lowercase form moves the other way \(−0\.306\-0\.306\); in elephant–tiger every form moves toward E, consistent with its negative transfer\. In the replay, the predicted sign of each form’s change relative toBBmatches in 11 of 12 form predictions at step 80 and in 8 of 12 at step 16; the overlap with the gradient of the preference margin is−0\.078\-0\.078for the capitalized form in sloth–lemur and\+0\.166\+0\.166to\+0\.244\+0\.244for the four forms in elephant–tiger\. Replaying Llama · FT · syllable · shrew–cuttlefish reproduces the start\-policy form margins to three decimals and matches all four signs at steps 16 and 80, 15 of 16 form predictions at step 80 over the four cells\. The cell whose choice changed has the smallest overlaps \(\+0\.002\+0\.002to\+0\.060\+0\.060\) and the single sign failure\.

The prior log\-probability of a pair’s names at the first position of an open question runs from−12\-12to−29\-29nats, and typicality in an animal list together with token count accounts forR2R^\{2\}0\.74 of it in Llama and 0\.44 in Qwen\. Along the path to a changed choice \(Fig\.[4](https://arxiv.org/html/2610.41301#S0.F4)c\), 24 cells reach one\-sided significance \(18 established\), 12 of the cells measured carry the shift on the first name token, 3 spread it to the direct\-answer form and 2 change the sampled choice\.

The open\-question evaluation passes the common\-component criterion in 21 cells \(19 Qwen, 2 Llama\); behaviour is read from sampled answers in twelve cells, the Llama cells with transfer among them\.

Completions that raise the names’ common component are atypical: their response correlates with per\-token base likelihood at−0\.38\-0\.38to−0\.43\-0\.43\. The self\-reinforcing sharpening is stronger under same\-model than under other\-family judges\.

Between 22 and 25% of base answers are refusals naming both animals \(“I don’t have a preference for either warblers or slugs”\), which a first\-name rule would score as a choice\.

The forms move independently within one student: in Llama · SP · number · warbler–slug the capitalized singular rises while the plural falls\.

Qwen · FT · syllable · otter–beaver, Qwen · SP · syllable · otter–beaver and Qwen · FT · number · sloth–lemur carry their shift on the first name token\. The prior log\-probability that their answer\-level noise follows is the names’ log\-probability at an open question’s first position\.

## Supplementary Note 9: The state of the optimization

Because the trait signal is small, the optimization state is reported cell by cell \(Extended Data Table[5](https://arxiv.org/html/2610.41301#Sx5.T5)\)\.

Both carriers stay valid\. At step 80 the mean validity of a cell \(Extended Data Table[5](https://arxiv.org/html/2610.41301#Sx5.T5)\) is 0\.980–0\.984 in Llama · FT · syllable, 0\.968–0\.975 in Llama · SP · syllable, 0\.991–0\.999 in Qwen · FT · syllable, 0\.937–0\.993 in Qwen · SP · syllable, 0\.998–0\.999 in both Llama number configurations and 0\.986–1\.000 in Qwen · FT · number, and it is exactly 1\.000 at every step in the six Qwen · SP · number cells; the lowest single run is listed per cell in the same table\. The Llama syllable cells start near 0\.53 and cross 0\.9 at steps 32–48\. An independently written implementation of each carrier rule, applied to the recorded completions, reproduces the logged validity at every recorded step to within 0\.03\. The two cells where one judge condition ends lower have non\-trait causes\. In the E\-judge runs of Qwen · SP · syllable · elephant–tiger, four of eight seed pairs end at 0\.75–0\.83: the policy has collapsed onto lists of English number words, and a list reaching “eleven” is rejected by the first\-token block for*elephant*\(*el*\), which accounts for 2,149 and 1,128 invalid completions against 36 and 14 format violations in the two logged seed pairs, while the T\-judge runs of the same seeds have 0\.000–0\.001 invalid; the concealment rule catches a coincidence, not a leak\. In the E\-judge run of Qwen · FT · number · jerboa–badger, one seed pair ends at 0\.862\. A logged run of the same seed reproduces the values \(0\.658 at step 64, 0\.862 at step 80\) and shows that the late invalid completions are all count violations: the policy converges on short arithmetic lists such as “0\.12, 0\.34, 0\.56, 0\.78”, below the five\-item minimum \(1,969 of the invalid completions in calls 48–79, with no character violations\)\.

Diversity collapses, as expected for GRPO: groups whose eight completions tie on reward, and so carry no policy\-gradient signal, rise in every cell\. At step 80 their share is 0\.04–0\.07 in Llama · SP · syllable, 0\.22–0\.39 in Llama · FT · syllable, 0\.09–0\.13 in Llama · SP · number, 0\.26–0\.46 in Llama · FT · number, 0\.30–0\.33 in Qwen · SP · number, 0\.47–0\.56 in Qwen · SP · syllable, 0\.56–0\.84 in Qwen · FT · syllable and 0\.57–0\.86 in Qwen · FT · number, a majority in Qwen with fine\-tuned judges \(Extended Data Fig\.[2](https://arxiv.org/html/2610.41301#Sx5.F2)\)\. The share of groups still carrying a signal falls below one half at steps 34–68 in the Qwen FT cells and 43–67 in Qwen · SP · syllable, and stays above one half through step 80 in all 24 Llama cells and the six Qwen · SP · number cells\. Across the 46 cells, that endpoint correlates with the policy’s entropy at step 40 at\+0\.63\+0\.63and with its starting entropy at\+0\.43\+0\.43; starting entropies differ by an order of magnitude \(3\.0 for Llama syllables, 0\.8 for Llama numbers, 1\.4 for Qwen syllables, 0\.10 for Qwen numbers\)\. Within one model and carrier, the route that pulls the policy further also collapses further: Qwen number cells reach KL 0\.46–0\.93 by step 40 under fine\-tuned judges and end with a signal share of 0\.14–0\.43, against KL 0\.07–0\.08 and 0\.67–0\.70 under persona judges\. The collapse does not confound the result: it partly reflects the starting distribution \(22–25% tied groups by step 40 in the Qwen · SP · number cells, from a starting entropy of 0\.10, with a KL ten times smaller than in any other configuration\), and established cells span final signal shares from 0\.14 \(Qwen · FT · number · jerboa–badger\) to 0\.96 \(Llama · SP · syllable · warbler–slug\)\.

Transfer accrues while the reward carries a signal and survives its collapse \(Extended Data Fig\.[2](https://arxiv.org/html/2610.41301#Sx5.F2)c\); transferring Llama cells reach half their finalSSbetween steps 32 and 64, when in\-training alignment forms\. Cells whose signal has collapsed can still accumulate afterwards \(Qwen · FT · syllable · otter–beaver gains a further 28% of its endpoint after reaching half of it, Qwen · FT · number · jerboa–badger 37%, Qwen · FT · number · warbler–slug 17%\) or decline slightly from a peak \(Qwen · SP · syllable · warbler–slug by 7% from its step\-32 peak, Qwen · FT · number · sloth–lemur by 27% from its step\-48 peak\)\. Declines are uncorrelated with final signal share \(r=−0\.03r=\-0\.03, 46 cells\), declines of comparable size occur where the signal never collapses, and the KL rises rather than falls from step 48 to 80 in exactly the collapsed cells \(rr−0\.73\-0\.73between signal share and the change in KL\)\.

Early Llama syllable reward is partly format: the valid/invalid indicator explains 0\.25, 0\.17, 0\.15, 0\.06 and 0\.05 of the within\-group reward variance over calls 0–7, 8–15, 16–31, 32–47 and 48–79 in Llama · FT · syllable · elephant–tiger, with the same profile in the other Llama syllable cells measured, and at most 0\.04 throughout in every Qwen cell and in the number carrier\. Format comes first and transfer after\. The Llama number cells, without a format phase, share that timing, so it marks when alignment forms\.

The KL penalty acts differently in the two students\. Settingβ\\betafrom 0\.04 to 0 in two Llama · SP · syllable cells, with everything else and the seeds unchanged, removes the mean reversion of the preference margin: the share of its variance carried by covariance between successive increments goes from−0\.81\-0\.81to−0\.14\-0\.14\(difference\+0\.67\+0\.67\[\+0\.38\+0\.38,\+1\.01\+1\.01\], bootstrapP<0\.00025P<0\.00025\)\. In Llama the penalty thus creates transient swings of the margin and pulls them back\. In two Qwen · SP · syllable cells the same change leaves that share unchanged \(\+0\.47\+0\.47against\+0\.54\+0\.54,PP0\.38\) and only enlarges the variance by a factor of 1\.7: in Qwen the penalty damps the movement without reversing it\. Consistently, the common movement of the two judges’ students in five of the six Llama · SP · syllable cells peaks around step 24 \(\+0\.28\+0\.28to\+0\.63\+0\.63\) and returns close to zero by step 80 \(elephant–tiger\+0\.583\+0\.583to\+0\.013\+0\.013\), whileSSis kept\.

In Llama number cells 92% of completions reach the 64\-token limit, so brevity is the strongest and most consistent reward component \(within\-group reward standard deviation 18\.6 against 8\.0 in Qwen\)\. AdamW normalizes each parameter’s step by its own running scale, so parameters dominated by this component move with the same sign in every seed; the result is a large common movement of both judges’ students and seed noise reduced to 0\.44 of what a random reward gives \(1\.01 in Qwen\)\. A one\-step probe built on this reproduces the observed first\-step ratio between the two students \(3\.7 against 3\.1–4\.5 for syllables, 3\.8 against 4\.2 for the persona\-judged number cells\)\. Across 37 configurations spanning 21 carriers, end\-of\-training KL follows the ratio of final to initial completion length \(ρ=−0\.75\\rho=\-0\.75,p<10−5p<10^\{\-5\}, holding within every student and route\), or the change in entropy where length is fixed\. The persona\-judged Qwen number cells are the extreme case: the carrier fixes the length, validity is 1\.000 from step 2 and the final KL is only 0\.089, yet the reward improves in all 96 runs \(mean\+8\.0\+8\.0, closing 69% of the reward gap\) and transfer accrues: the trait needs very little of the KL budget\.

GRPO’s diversity collapse removes the reward’s policy\-gradient signal in 16 of the 46 cells before step 68, by which point most transfer has accrued and stays\.

## Supplementary Note 10: How far the first\-order account reaches

The first\-order expression at the start policy holds for a short window\. The push accumulated to steps 8 and 16 tracks the increments ofSSover those intervals \(rank correlation\+0\.20\+0\.20,p=0\.018p=0\.018;\+0\.21\+0\.21,p=0\.015p=0\.015\), and from step 16 onward the relation disappears and turns slightly negative \(−0\.19\-0\.19over steps 16–32\)\. Over the whole run, the push on the logged training samples does not predict the outcomes of 102 seed pairs in 35 cells: its sign matches that ofS80S\_\{80\}in 55 of 102 pairs \(p=0\.24p=0\.24\) and in 40 of the 69 with\|S80\|\|S\_\{80\}\|above 0\.01 \(58%,p=0\.11p=0\.11\), with rank correlation−0\.01\-0\.01; late\-training alignment, its correlation form, does no better \(51 of 102, rank\+0\.08\+0\.08\), and over whole cells the counts are 19 and 20 of 35\.

Within a cellS80S\_\{80\}varies by±0\.1\\pm 0\.1or more across seeds, and this variation dominates the design: 96–98% of the variance ofSSis seed variation within a trait pair, and the training that produces the transfer also makes the preference evaluation 4\.8 to 6\.5 times noisier than for pairs the run never touched\. Three explanations of the unpredictability fail\. A measurement gap: the push from all 80 calls, about 160,000 valid samples per cell, gives the same pair\-level correlation, negative in two of three cells \(−0\.40\-0\.40,−0\.28\-0\.28and−0\.34\-0\.34betweenS80S\_\{80\}and the full push\)\. Optimizer geometry: applying AdamW to the gradients of the recorded batches at the start policy over all 80 steps predicts no better than the Euclidean replay \(correlation withS80S\_\{80\}\+0\.87\+0\.87,\+0\.33\+0\.33and−0\.30\-0\.30in the three cells against\+0\.94\+0\.94,\+0\.17\+0\.17and\+0\.06\+0\.06; sign agreement 2, 3 and 1 of 4 against 3, 3 and 3\), although in the first step the first\-token share is 76–85% of the Euclidean gradient and only 0\.6–13% of the AdamW step\. A decisive early step: the correlation betweenSSat stepttandS80S\_\{80\}is at chance fort≤8t\\leq 8in every cell, andSSover the first eight steps is unrelated toS80S\_\{80\}; in the persistent cells it exceeds 0\.8 by steps 24–48\.

The seed deviation is a bounded fluctuation that saturates and partly reverts\. Over 106 seed pairs in 37 cells, centred within cell, the spread ofSSacross seeds grows from 0\.012 at step 8 to 0\.028 at step 32 and then flattens \(0\.032 at step 48, 0\.034 at step 80\); the deviation a pair has accumulated by step 32 explains less than a quarter of its endpoint deviation \(rr\+0\.47\+0\.47, against\+0\.74\+0\.74by step 48 and\+0\.90\+0\.90by step 64\)\. Deviations already opened are pulled back: the deviation at step 32 correlates at−0\.37\-0\.37with the increment that follows, and consecutive increments correlate at−0\.07\-0\.07to−0\.18\-0\.18\. A seed pair’sS80S\_\{80\}is the cell mean plus one draw of a fluctuation of roughly 0\.03\. Across cells, a cell’s spread across seeds follows the spread of the two judges’ common movement \(rr\+0\.79\+0\.79;R2R^\{2\}0\.74 on the log scale\), the student model adds only 0\.04, and baseline preference and name token count explain almost nothing\. The seed component has a fixed shape: across name pairs it lies on one axis of the preference margin \(the first principal component takes 56–78% of its variance, against 12–25% for random directions\), and a step changes only its sign along that axis\. A single seed pair therefore carries almost no information about a cell, whose result needs eight or more pairs\. The replay also locates where the start\-policy prediction stops: for the two cells whose mean movement it matches in sign, predicted and actualΔ​B\\Delta Bkeep a constant ratio through step 16 \(0\.06–0\.14\) and then diverge, the actual movement plateauing while the linear prediction keeps growing, so the ratio falls to 0\.03–0\.04 by step 80; the between\-cell correlation peaks at step 8 \(\+0\.55\+0\.55to\+0\.70\+0\.70\) and declines thereafter\. Seed\-pair outcomes, which form at steps 24–48, form where the start\-policy gradient no longer describes the movement\.

What fails is the frozen gradient, not the first\-order account\. Expanding the preference margin at saved checkpoints—its gradient atθt\\theta\_\{t\}against the displacement realized betweenθt\\theta\_\{t\}andθt\+1\\theta\_\{t\+1\}—recovers the increments ofSS, in sign in 59 of 60 intervals \(r=\+0\.93r=\+0\.93; twelve seed pairs in four cells spanning both students, routes and carriers\)\. The expression needs intervals short against the movement inside them: in the four pairs cut at steps 2, 4, 8, 16, 32 and 48 the correlation is\+0\.98\+0\.98over their 28 intervals and the accumulated total is 0\.96 to 1\.44 times the measured one, whereas a single interval over the first 16 steps overstates it by up to seventeen times in Llama · SP · number · sloth–lemur, the cell whose policy moves furthest early; cutting that interval into four brings its total to 1\.13 and 1\.44 of the measured value in the two pairs cut that way\. The start policy cannot carry the alignment, which turns as the policy moves; pair\-to\-pair differences are seed path, not a quantity the start policy determines\.

Across the grid the trait pair is the largest single factor inS80S\_\{80\}\(R2R^\{2\}0\.208\), and model, route and carrier act only through their three\-way combination \(R2R^\{2\}0\.281 for the eight configurations against 0\.117 for the three factors added\); configuration and trait pair together reach 0\.484, and the remainder follows absorption \(r=\+0\.39r=\+0\.39\)\. Crossing the significance threshold, however, follows detection noise more than the predicted effect: an absorption\-based classifier reaches a leave\-one\-out accuracy of 0\.57, below the 0\.61 of calling no cell established\. The account predicts transfer as a continuous quantity across cells, trait pairs and judge instances, not whether a cell is established, nor the individual seed\.

## Supplementary Note 11: How much of the push the optimizer realizes, and where

We decomposed the trained displacement of eight cells \(four per route, eight seed pairs each\) along each route’s first GRPO direction, the between\-judge update at the start policy \(Supplementary Methods\)\. That direction is stable across candidate pools \(cosine 0\.92–1\.00 between independent pools\) and nearly orthogonal to the fine\-tuned judges’ own weight difference \(cosine about 0\.002\), so the student is not drawn toward the judge’s weights\. Each student follows its own route’s direction: FT students move along the FT direction and not the SP one \(cosine ofΔ\\DeltawithvFTv\_\{\\mathrm\{FT\}\}\+0\.057\+0\.057, withvSPv\_\{\\mathrm\{SP\}\}\+0\.005\+0\.005, over three of the four FT cells\), and SP students along the SP direction \(\+0\.040\+0\.040against\+0\.009\+0\.009, over the four SP cells\)\. The displacement is a small rotation in a large parameter space, and its projection on that direction moves the preference margin: in every same\-model cell decomposed, the component in the plane of the two directions movesBBby more than the whole displacement does\.

That component overshoots the transfer, and the rest of the displacement takes most of it back in layers 0–3\. In six cells spanning both routes, students and carriers \(Llama · FT · syllable · sloth–lemur, Llama · FT · syllable · warbler–slug, Qwen · FT · syllable · jerboa–badger, Qwen · FT · number · warbler–slug, Llama · SP · syllable · warbler–slug and Qwen · SP · syllable · warbler–slug\), this early\-layer remainder is a nearly fixed fraction of the component \(slope−0\.82\-0\.82through the origin,r=−0\.998r=\-0\.998\), as an accounting of the endpoint in which the direction component varies far more than the transfer\. What reaches the preference margin is the part left over, with a small contribution from the other layers \(−0\.010\-0\.010to\+0\.051\+0\.051\): 0\.15 to 0\.31 of the direction component in four of the six cells, 0\.05 in Qwen · FT · syllable · jerboa–badger and 0\.64 in Llama · FT · syllable · sloth–lemur, a median of 0\.19\. In Qwen · SP · syllable · warbler–slug, for example, the direction component movesSSby\+1\.297\+1\.297in fp32, the remainder in layers 0–3 by−1\.088\-1\.088, and the transfer is\+0\.217\+0\.217\. The two exceptions, Llama · SP · number · sloth–lemur and Qwen · SP · number · elephant–tiger, the persona\-judged number cells with the weakest emission and absorption decomposed, have a small component that is not taken back, and Llama · SP · number · sloth–lemur transfers mainly through a movement of layers 4–7 nearly orthogonal to the direction \(cosine 0\.045\)\. Summed component effects reproduce the full displacement’s effect to within 0\.4–7\.9% of the direction component per cell, and to 0\.3–5\.8% of the endpoint effect in seven of the eight cells; the exception, Qwen · FT · syllable · jerboa–badger, has the smallest endpoint effect \(\+0\.013\+0\.013\), of the same order as its residual\.

The shortfall sits where the alignment lives \(Supplementary Table[S1](https://arxiv.org/html/2610.41301#Sx16.T1)\)\. In layers 0–3 the student follows the direction at 0\.23 to 0\.56 of the network\-wide coefficient, against 1\.4 to 3\.2 in the deeper layers, while nearly all of the direction’s effect on the preference margin sits in those same layers \(\+1\.12\+1\.12of\+1\.30\+1\.30in Qwen · SP · syllable · warbler–slug,\+0\.66\+0\.66of\+0\.78\+0\.78in Qwen · FT · number · warbler–slug,\+0\.17\+0\.17of\+0\.19\+0\.19in Llama · FT · syllable · sloth–lemur\)\. The early\-layer gradient is concentrated, its top 1% of coordinates carrying 86–96% of its energy in layers 0–3 against 37–90% in deeper layers, and these are the coordinates of the first\-token path on which absorption runs\. AdamW, dividing each coordinate’s step by its own running scale, brings the step toward the sign of the gradient: in layers 0–3 the displacement aligns withsign⁡\(v\)\\mathrm\{sign\}\(v\)2\.3 to 4\.4 times better than withvv, against 1\.0 to 2\.7 times in the deeper layers\. The optimizer therefore moves the few coordinates that carry the raw alignment much less than the raw gradient asks; a direction component computed with network\-wide coefficients overstates the early\-layer movement accordingly, and the early\-layer remainder is that overstatement with its sign reversed\. The pattern holds in both routes and both students and is strongest in the two Qwen cells, whose early\-layer gradient is the most concentrated\. Realized transfer spreads over the whole depth:\+0\.012\+0\.012to\+0\.041\+0\.041in each four\-layer group of Qwen · SP · syllable · warbler–slug,\+0\.018\+0\.018to\+0\.056\+0\.056in the first five groups of Qwen · FT · number · warbler–slug with the last two slightly negative, and\+0\.054\+0\.054,\+0\.031\+0\.031,\+0\.023\+0\.023and\+0\.018\+0\.018in the four groups of Llama · FT · syllable · sloth–lemur, where the early layers carry a larger share\.

Table S1 \|:Layer\-wise optimizer geometry\.For one cell of each route and student: the following ratio \(least\-squares coefficient of the displacementΔ\\Deltaon the first GRPO direction within a four\-layer group over the network\-wide coefficient\), the sign alignment ratiocos⁡\(Δ,sign​v\)/cos⁡\(Δ,v\)\\cos\(\\Delta,\\mathrm\{sign\}\\,v\)/\\cos\(\\Delta,v\)and the share of the squared norm ofvvin the top 1% of a group’s coordinates, in layers 0–3 and across the deeper groups \(two seed pairs per cell\)\.Hence blocking the first\-token path, which the optimizer barely follows, leaves transfer in place, and start\-policy absorption predicts transfer across cells but not its size: a median fifth of the push it measures appears in the endpoint of the six cells that show the cancellation, while the share the early layers give back is the nearly constant part\. Within a cell, where transfer is strong the direction component varies with the seed in step with the outcome \(correlation withS80S\_\{80\}across seed pairs\+0\.60\+0\.60to\+0\.85\+0\.85in Llama · FT · syllable · sloth–lemur, Qwen · SP · syllable · warbler–slug and Qwen · FT · number · warbler–slug\) and every seed pair moves along the direction; where it is weak, the direction component is positive in most pairs while the remainder, equal in size or larger and changing sign between seeds, tracks each pair’s sign \(\+0\.4\+0\.4to\+0\.9\+0\.9\)—the weight\-level form of the seed variation above\.

## Supplementary Note 12: Judges of other models, stage by stage

The cross\-family condition is interpretable on the main grid’s terms: recipe and precision are identical, the share of invalid completions is similar \(0\.02–0\.20 against 0\.01–0\.19\), and the judges’ trait differences are as reliable \(eight\-instance reliability 0\.91–0\.99 on the SP route and 0\.79–0\.93 in the FT syllable cells\)\. The per\-cell checks for length confounding, distance travelled and degeneration of the open\-question evaluation flag two cells, both through a trait difference that partly tracks completion length; in Qwen · FT · syllable · jerboa–badger part of the trait difference is a per\-token likelihood offset of the Llama jerboa–badger fine\-tuned judges \(−0\.067\-0\.067per token\), which the same judges also show on the Llama student’s pool in the main grid \(−0\.065\-0\.065\), and with that component removed its absorption is\+0\.127\+0\.127\.

On the SP route emission is positive in 23 of 23 cells under cross\-family judges, with an unchanged mean \(\+0\.296\+0\.296under same\-model judges,\+0\.310\+0\.310under cross\-family judges\), and its order across cells within a configuration is kept \(rr\+0\.67\+0\.67\), as expected of a quantity that is 46% a property of the trait pair and 30% of the student’s candidate pool, with 7% due to which of the two models the judge is and 1% to whether it is the student’s own model \(balanced decomposition over 44 combinations; judge\-instance noise 1 percentage point\)\. The policy travels at least as far: end\-of\-training KL is 1\.5 to 2\.0 times that of the same\-model run in the syllable cells \(4\.9 in Qwen · SP · syllable · jerboa–badger, whose same\-model run stops moving after step 24\) and 1\.3 to 1\.5 times in the Llama number cells, because the other family’s judge assigns the student’s completions a lower likelihood and the reward keeps rising from further below\. The cross\-family judge’s trait difference correlates with the same\-model judge’s at 0\.05 after correction for unreliability, and the part of absorption inherited from the same\-model judge \(equation \([15](https://arxiv.org/html/2610.41301#Sx2.E15)\)\) is\+0\.005\+0\.005for the Llama student and\+0\.002\+0\.002for the Qwen student\. In the Llama student the mean size of absorption falls from 0\.209 to 0\.040; in the Qwen student most of the size remains \(0\.133 to 0\.093\), but its sign agrees with the same\-model sign in 5 of 11 cells\. The meanS80S\_\{80\}over the same seed pairs falls from\+0\.013\+0\.013to\+0\.002\+0\.002\.

On the FT route emission is present in 22 of 23 cells with an unchanged mean \(\+0\.126\+0\.126and\+0\.132\+0\.132\), but its order across cells within a configuration is barely kept \(rr\+0\.10\+0\.10\), because here 50% of the variance ofΛ\\Lambdais specific to the combination of a judge set with a candidate pool, against 35% for the trait pair and 12% for the pool\. Qwen · FT · number · warbler–slug emits at\+0\.184\+0\.184under its own judges and\+0\.049\+0\.049under cross\-family judges, and the one cell without emission under them, Qwen · FT · number · shrew–cuttlefish \(−0\.002\-0\.002\[−0\.011\-0\.011,\+0\.008\+0\.008\]\), emits at\+0\.064\+0\.064under its own\. The policy travels 0\.6 to 0\.8 times as far in the Qwen syllable cells, 0\.8 to 0\.95 times in the Llama syllable cells and 1\.1 to 1\.2 times in the Llama number cells\. The reliability\-corrected correlation between the two judges’ eight\-instance trait differences is 0\.24, between the two training instances 0\.03, and the inherited part of absorption is−0\.003\-0\.003for the Llama student and\+0\.011\+0\.011for the Qwen student\. Mean absorption size falls from 0\.170 to 0\.066 in Llama and from 0\.100 to 0\.084 in Qwen, whose cross\-family sign agrees with the same\-model sign in 7 of 11 cells\. The meanS80S\_\{80\}over the same seed pairs falls from\+0\.014\+0\.014to\+0\.003\+0\.003\.

Pooled over all 46 cells, the cross\-family mean is\+0\.0021\+0\.0021\[−0\.0010\-0\.0010,\+0\.0052\+0\.0052\] weighting cells by precision and\+0\.0024\+0\.0024\[−0\.0029\-0\.0029,\+0\.0077\+0\.0077\] weighting them equally, whereas the same\-model mean in those cells is\+0\.0161\+0\.0161and\+0\.0138\+0\.0138, outside both cross\-family intervals\. Paired over the same cells and seeds, the same\-model effect exceeds the cross\-family effect by\+0\.0100\+0\.0100\[\+0\.0062\+0\.0062,\+0\.0138\+0\.0138\] with cells weighted by the precision of their paired difference and by\+0\.0114\+0\.0114\[−0\.0004\-0\.0004,\+0\.0233\+0\.0233\] with cells and seed pairs resampled with equal weight\.

The judge term of absorption behaves differently in the two students\. In Llama it collapses under the cross\-family judge, from 0\.209 to 0\.028 on the SP route and from 0\.113 to 0\.037 on the FT route, which accounts for the fivefold loss of absorption on the SP route, where the first\-token path sets its sign \(9 of 12 cells\)\. In Qwen the first\-token path is closed on the student side whatever the judge: the gradients of the twelve names lie on the direction common to animal names \(cosine 0\.84–0\.95\), so the difference between a pair’s names barely projects on it \(\|vB⋅μ^\|≤0\.023\|v\_\{B\}\\cdot\\hat\{\\mu\}\|\\leq 0\.023, against 0\.017–0\.139 in Llama\)\. There the cross\-family judge term is as large as the own judge’s \(0\.095 against 0\.119 on the FT route, 0\.094 against 0\.080 on the SP route\), and absorption keeps its size through the remaining paths\.

In the syllable carrier every judge—the student’s own, the other family’s or either 8B model—prefers strings whose letters overlap its own animal’s name \(correlation of its letter coefficients with the name letters\+0\.09\+0\.09to\+0\.52\+0\.52\), so this part is shared: the feature parts of the own\-model and cross\-family trait differences correlate at\+0\.15\+0\.15to\+0\.21\+0\.21and supply\+0\.02\+0\.02to\+0\.06\+0\.06of an alignment of\+0\.03\+0\.03to\+0\.17\+0\.17\. In the number carrier no such cue exists: the judges’ preference for number size is near zero, the feature parts do not agree \(−0\.27\-0\.27to\+0\.25\+0\.25\) and cross\-family alignment is−0\.10\-0\.10to\+0\.004\+0\.004\. With the other model as student, the cross\-family judge’s trait difference correlates with that model’s response of the preference margin at 0\.17 on average over the 46 cells, as strongly as the student’s own judge does with the student’s \(0\.17\), but the two models’ responses to the same candidates barely agree \(\+0\.04\+0\.04on average;\+0\.06\+0\.06to\+0\.16\+0\.16in the syllable cells, mostly through the letters, and−0\.06\-0\.06to\+0\.06\+0\.06in the number cells\)\. Cross\-family absorption in the syllable cells comes mostly through the shared letters \(\+0\.045\+0\.045to\+0\.074\+0\.074;−0\.004\-0\.004in Qwen · SP · syllable\), which across cells predictsSSof 0\.003 to 0\.005\.

Across the 22 Qwen cells, start\-policy absorption predicts the cross\-familyS80S\_\{80\}atrr\+0\.40\+0\.40\(p=0\.06p=0\.06\), against\+0\.59\+0\.59\(p=0\.004p=0\.004\) for the same\-model relation in the same cells; the relation is strong in the syllable carrier \(rr\+0\.84\+0\.84,p=0\.002p=0\.002, 10 cells\) and\+0\.27\+0\.27in the number carrier \(12 cells\)\. Over both students it is\+0\.50\+0\.50in the 22 syllable cells \(p=0\.018p=0\.018\) and\+0\.06\+0\.06in the 24 number cells, and in Llama cross\-family absorption sits near its floor and does not orderSS\(rr−0\.05\-0\.05over 24 cells\)\. In the Qwen syllable cellsSStracks the carrier\-feature part of cross\-family absorption \(rr\+0\.82\+0\.82,p=0\.004p=0\.004\), not the remainder \(\+0\.53\+0\.53, not significant\)\. Seed noise does not account for the number\-carrier null: with the observed variance over seed pairs, absorption predictsrr\+0\.84\+0\.84\(90% range\+0\.67\+0\.67to\+0\.94\+0\.94\) in the Qwen number cells, against the observed\+0\.27\+0\.27\. The first GRPO step agrees\. On the Qwen student’s syllable warbler–slug candidates, the cross\-family first step moves the per\-token answer margin in the trait direction at a fifth to two thirds of the same\-model step \(\+5\.91\+5\.91and\+1\.74\+1\.74along the cross\-family FT and SP directions, against\+9\.34\+9\.34and\+8\.50\+8\.50along the same\-model ones\)\. In the number cells the margin moves by a comparable amount with a sign set by the judge \(Qwen number warbler–slug:\+6\.50\+6\.50and\+2\.61\+2\.61along the same\-model directions against−1\.89\-1\.89and−4\.42\-4\.42; elephant–tiger:\+1\.02\+1\.02and\+3\.31\+3\.31against−0\.98\-0\.98and\+2\.32\+2\.32\)\.

Training does not supply the missing alignment\. In Qwen · FT · number · warbler–slug and Qwen · SP · number · elephant–tiger, every group scored during cross\-family training was recorded \(16 runs each\) and the completions of ten evenly spaced calls rescored with two independent instances of the student’s own judges\. In groups still carrying a learning signal, the advantage the cross\-family judge assigned is uncorrelated with the own\-model trait difference of the second instance \(−0\.033\-0\.033, positive in 7 of 16 runs;−0\.030\-0\.030, 6 of 16\), early \(calls 0–26\) and late \(calls 53–79\) alike, whereas the advantage the first own\-model instance assigns to the same completions is aligned with it \(\+0\.042\+0\.042, 12 of 16,t=2\.1t=2\.1;\+0\.059\+0\.059, 15 of 16,t=5\.0t=5\.0\)\. The share of groups carrying any signal falls from 0\.89–0\.97 to 0\.40 under both judges\.

In the number carrier the push is not only unaligned but unrealized\. In Qwen · FT · number · warbler–slug, along the start policy’s gradient of the preference margin, the cross\-family between\-judge difference on the recorded training completions pushes the margin against the trait in 13 of 16 runs \(correlation−0\.106\-0\.106,tt−4\.3\-4\.3, early and late alike\), as consistently as the own\-model difference pushes it toward the trait on the same completions \(\+0\.105\+0\.105, 16 of 16\); weighting calls by how far the policy moved leaves the cross\-family value unchanged \(−0\.108\-0\.108\)\. Yet the cross\-familyS80S\_\{80\}of this cell is\+0\.005\+0\.005, and across its eight seed pairsSSdoes not follow the size of the push \(rr−0\.32\-0\.32\)\. The candidates’ responses of the preference margin at the end of training are uncorrelated with those at the start \(\+0\.009\+0\.009, averaged over runs\), and along the end\-of\-training gradient the cross\-family push reverses sign \(\+0\.046\+0\.046, 12 of 16\), while the own\-model difference keeps its sign \(\+0\.123\+0\.123, 14 of 16\)\. In same\-model training of the same cells, recorded and rescored the same way, the own judge’s difference pushes the margin toward the trait along both the start and the end\-of\-training gradient \(Qwen · FT · number · warbler–slug\+0\.149\+0\.149and\+0\.036\+0\.036, positive in 15 and 12 of 16 runs; Qwen · SP · number · elephant–tiger\+0\.172\+0\.172and\+0\.149\+0\.149, 16 and 15 of 16\), and the gradient turns less than under the cross\-family judge \(correlation between start and end responses\+0\.057\+0\.057and\+0\.169\+0\.169, against\+0\.009\+0\.009and−0\.031\-0\.031\)\. In Qwen · SP · number · elephant–tiger the cross\-family push stays near zero along both gradients \(\+0\.031\+0\.031and\+0\.041\+0\.041\)\.

Answer\-level effects under cross\-family judges appear in four of the 46 cells, two in the judge’s direction \(Qwen · FT · number · sloth–lemur, Qwen · SP · number · sloth–lemur\) and two against it \(Llama · SP · number · jerboa–badger, Llama · SP · number · warbler–slug\)\. Unlike the same\-model condition, the open\-question evaluation of the Llama syllable cells does not degenerate under cross\-family judges \(common component−0\.25\-0\.25to\+0\.31\+0\.31\), so behaviour is not compared between the two conditions in that configuration\.

An 8B judge’sΛ\\Lambdahas a lower bound above zero in 9 to 12 of the cells of every route–student combination, with means of\+0\.10\+0\.10to\+0\.41\+0\.41against\+0\.12\+0\.12to\+0\.37\+0\.37for the student’s own judges \(Extended Data Table[6](https://arxiv.org/html/2610.41301#Sx5.T6)\)\. Its trait difference correlates with the own\-model judge’s at 0\.01 to 0\.12 for the judge of the student’s family and 0\.02 to 0\.12 for the other family\. In the Llama student its judge term is 0\.038 to 0\.080 under either 8B judge, against 0\.113 and 0\.209 under the student’s own judges; absorption is 0\.047 to 0\.144 in size, with the sign of the own model in 6 to 10 of 11 or 12 cells\. An 8B judge places the trait where its own model’s preference margin responds: its trait difference correlates with the 8B model’s response at 0\.22 \(same family\) and 0\.23 \(other family\) on average over the 46 cells, as strongly as the student’s own judge does with the student’s \(0\.17\)\. The 8B models’ responses agree with the student’s only slightly,\+0\.12\+0\.12for the student’s family and\+0\.08\+0\.08for the other, against\+0\.04\+0\.04for the cross\-family 1B model: sharing the family triples the overlap but leaves it at about an eighth of the student’s own\. The overlap is largest in the syllable cells \(\+0\.13\+0\.13to\+0\.27\+0\.27, two thirds of it through letter content\) and close to zero in the number cells \(0\.00 to\+0\.09\+0\.09\)\. Trained on eight established cells with both 8B judges on the same eight seed pairs, the same\-seed meanS80S\_\{80\}falls from\+0\.041\+0\.041under the student’s own judges to−0\.007\-0\.007under the same\-family 8B judge and\+0\.012\+0\.012under the other\-family one; the closest of the 16 combinations, Llama · SP · syllable · warbler–slug with the other\-family judge \(\+0\.048\+0\.048, 7 of 8 seed pairs\), moves in the judge’s direction without passing the false\-discovery threshold \(q=0\.13q=0\.13\)\. One answer\-level effect appears, against the judge’s direction\.

## Supplementary Note 13: Cross\-family judges in the weights

In the weights, the cross\-family component along the route’s direction falls to a small fraction of the same\-model value, all taken back in the early layers: in the eight cells decomposed under both conditions it falls from\+0\.364\+0\.364to\+0\.053\+0\.053on average and the students’ alignment with the FT direction from\+0\.031\+0\.031to\+0\.004\+0\.004, and the early\-layer remainder takes back all of what is left \(−0\.054\-0\.054against\+0\.053\+0\.053\), against three quarters in the same\-model students \(−0\.270\-0\.270against\+0\.364\+0\.364\)\. Decomposed on their own judge’s first\-step direction, cross\-family students follow it with a sign that changes between seeds in both carriers: 3 of 8 seeds are positive in Qwen · SP · number · elephant–tiger \(mean−0\.010\-0\.010\), 5 of 8 in Qwen · FT · number · warbler–slug \(\+0\.013\+0\.013\) and 5 of 8 in Qwen · SP · syllable · warbler–slug \(\+0\.014\+0\.014, with a mean effect on the preference margin along the direction of\+0\.13\+0\.13\), whereas the same\-model student follows its own direction in 8 of 8 seeds in Qwen · SP · number · elephant–tiger \(\+0\.026\+0\.026to\+0\.049\+0\.049\) and at\+0\.121\+0\.121in Qwen · FT · number · warbler–slug\. A push a fifth to two thirds the own\-model size therefore does not outweigh each seed’s own movement: in Qwen · SP · syllable · warbler–slug the same\-model students share a common direction of movement \(mean cosine between two seeds’ weight changes\+0\.14\+0\.14, positive in 22 of 28 pairs\), whereas the cross\-family students share none \(−0\.005\-0\.005, 13 of 28\)\. The alignment of the syllable cells appears in the mean over seeds, and the agreement between absorption andSSacross cells is carried by cell means rather than by every run\. AdamW’s sign\-based concentration in the early layers is the same in both conditions \(cos⁡\(Δ,sign​v\)/cos⁡\(Δ,v\)\\cos\(\\Delta,\\mathrm\{sign\}\\,v\)/\\cos\(\\Delta,v\)4\.1 against 4\.5\), so it does not account for the difference\. Llama · SP · number · sloth–lemur, whose same\-model transfer runs through layers 4–7 outside the direction, moves those layers the other way under the cross\-family judge \(−0\.021\-0\.021against\+0\.048\+0\.048\)\. The cross\-family student is trained and travels as far, but not along a direction that is fixed across seeds or to which its preference margin responds\.

## Supplementary Note 14: Dispositional cells

Emission in the twelve dispositional cells \(Λ\\Lambda\+0\.15\+0\.15to\+0\.64\+0\.64\) lies within the range of the main grid’s SP cells \(\+0\.06\+0\.06to\+0\.83\+0\.83\)\. The three cells least aligned with the student’s preference margin \(absorption−0\.070\-0\.070,\+0\.081\+0\.081and\+0\.069\+0\.069, all Llama\) have small effect sizes \(dd−0\.08\-0\.08,\+0\.04\+0\.04and−0\.00\-0\.00at step 16\)\. Across the twelve cells absorption and transfer correlate at\+0\.32\+0\.32to\+0\.46\+0\.46from step 8 to step 32 and at\+0\.14\+0\.14at step 80\. At step 80 the six established cells haveqqat most 0\.026 and the other six at least 0\.383\. Harsh–gentle on both Llama carriers moves with its alignment \(dd\+0\.35\+0\.35at step 16\), and harsh–gentle on the Qwen syllable carrier carries the largest transfer in the dispositional grid \(dd\+1\.39\+1\.39at step 64\)\. Llama · SP · number · persuasive–neutral transfers at six of nine steps \(ddup to\+0\.55\+0\.55\), but at baseline at most one item in ten is answered with the T disposition\. In Qwen · SP · number · harsh–gentle transfer forms late \(steps 48 to 80\) while the forced choice moves early \(steps 16 and 24\)\. Where the implied choice moves with the judge, the two students choose differently on 3\.4–4\.0% of items in Qwen · SP · syllable · rude–polite \(steps 32 to 80\) and on 0\.4–0\.8% in Qwen · SP · syllable · harsh–gentle \(steps 16 to 80\)\. The choice moves against the judge in Qwen · SP · number · persuasive–neutral, following that cell’s reversal \(−0\.62\-0\.62to−0\.81\-0\.81percentage points\), and in Llama · SP · number · harsh–gentle at step 24 \(−0\.75\-0\.75\)\.

The lowest lower bound of emission across the twelve dispositional cells is\+0\.14\+0\.14\. As in the main grid, alignment gives the direction in which a cell begins rather than the size it reaches\.

The two cells established against the judge at step 80 reverse during training: Qwen · SP · number · persuasive–neutral is with the judge at steps 8 and 16 and against it by step 64, and Qwen · SP · number · rude–polite is established against it from step 48\. Both lie in the Qwen number configuration, whose pair values do not settle in the main grid either\.

## Supplementary Note 15: Endpoint, judge instance and the scale of the preference margin

In three cells half of the final transfer accrues at step 48 or later\. Training Llama · FT · syllable · sloth–lemur for 160 steps at a constant learning rate keeps the sign \(six of eight seed pairs in the judge’s direction at step 80 and at step 160\), whileSSpeaks near step 48 \(\+0\.100\+0\.100\) and falls to\+0\.018\+0\.018by step 160; the main grid’s learning rate decays linearly to zero at step 80, so its endpoint is read after that peak\. In the Qwen number cells seed\-pair values do not settle: the correlation between a pair’sSSat step 64 and at step 80 falls as low as−0\.37\-0\.37\(Qwen · SP · number · jerboa–badger\), against 0\.73–0\.99 in the other cells\.

Training under the judge instances other than the training instance bounds the choice of training instance\. In Llama · FT · syllable all seven other judges agree in direction \(mean\+0\.034\+0\.034, 40 of 42 judge–trait\-pair combinations positive\); Llama · FT · number shows no transfer averaged over all eight judge instances \(\+0\.001\+0\.001, 24 of 48\), although within trait pairs it follows each judge’s absorption; in Qwen · FT · number, shrew–cuttlefish and warbler–slug are positive under all four judges tested and sloth–lemur only under the training instance\.

For the same update the Llama preference margin moves about 2\.5 times as much as Qwen’s, and the gap remains per unit of KL on the evaluation items \(3\.25 times\), so it is a property of the preference margin and not of stronger transfer\. It is accounted for by larger downstream gain in Llama’s early layers and by Qwen3 sending 56–83% of an update into eight massive channels that the final normalization largely absorbs\. Forward\-pass precision in training changes the estimate by less than eight seed pairs resolve: Llama · FT · syllable · sloth–lemur gives\+0\.033\+0\.033under fp32 against\+0\.073\+0\.073under bf16\.

For scale,S=0\.035S=0\.035means the T\-to\-E name likelihood ratio on the preference\-evaluation templates differs between runs by a factore2​S≈1\.07e^\{2S\}\\approx 1\.07, not seven percentage points in sampled choice\.

## Supplementary Methods

### Training

Training uses TRL[von Werra et al\. \(2020\)](https://arxiv.org/html/2610.41301#bib.bib17)GRPO with the DAPO loss[Yu et al\. \(2025a\)](https://arxiv.org/html/2610.41301#bib.bib18)and group\-level reward scaling[Liu et al\. \(2025\)](https://arxiv.org/html/2610.41301#bib.bib38): eight generations per prompt, 32 prompts \(256 completions\) per step, gradient accumulation 32, 80 steps, at most 64 completion tokens, KL penaltyβ=0\.04\\beta=0\.04to the starting policy[Ziegler et al\. \(2019\)](https://arxiv.org/html/2610.41301#bib.bib29), AdamW[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2610.41301#bib.bib41)\(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, no weight decay\), gradient\-norm clipping 1\.0, learning rate10−610^\{\-6\}decaying linearly to zero at step 80, fp32 master weights and bf16 forward pass\. The reward \(equation \([4](https://arxiv.org/html/2610.41301#Sx2.E4)\)\) has no student\-likelihood term; the policy loss covers the whole completion\.

### Trajectory measurements

Trajectory runs repeat a cell’s seed with a hook recording every prompt, completion and reward without changing training \(recorded reward within4\.4×10−44\.4\\times 10^\{\-4\}of the judge’s score;SSequal to the hook\-free run’s to four decimals\); the state of training is in Extended Data Table[5](https://arxiv.org/html/2610.41301#Sx5.T5)and Extended Data Fig\.[2](https://arxiv.org/html/2610.41301#Sx5.F2)\. The in\-training push applies equation \([6](https://arxiv.org/html/2610.41301#Sx2.E6)\) to the recorded samples \(half the between\-judge difference of standardized logged reward times each candidate’s start\-policykk\); in\-training alignment is its correlation form\.

Along\-the\-run accounting saves twelve seed pairs at steps 16, 32, 48 and 80 and pairs the gradient of the preference margin at each saved policy \(as forkk, margin in fp32\) with the update to the next saved policy\. Four pairs are also saved at steps 2, 4 and 8 from a second run of the same seed \(step\-80 margin equal to five decimals\); all 60 intervals enter the analysis\. Replay reconstructs the GRPO policy gradient from recorded samples at the fixed start policy, applies the AdamW update or the clipped Euclidean gradient, and accumulates the first\-order change inBBand each direct\-answer form margin, omitting the KL term \(zero at the reference point\)\.

### Where absorption runs

Path shares come from fixing a state and recomputing absorption\. Fixing the first\-token state at the inputs of Llama layers 1 and 2 accounts for 74–97% of absorption in the cells measured; in Qwen syllable cells 98–99% of the gradient passes through it\. About 80% of absorption arises in layers 0–3\. The massive\-activation neuron[Sun et al\. \(2024\)](https://arxiv.org/html/2610.41301#bib.bib14)carrying much of it is neuron 1417 of layer 1 in Llama \(40–51% of absorption through its output column\) and neuron 1821 of layer 2 in Qwen\.

### Weight\-level decomposition

Eight cells, four per route, are decomposed over the 16 students of eight seed pairs at step 80\. For a seed pair,

Δ=12\(θT,80−θE,80\),θmid=12\(θT,80\+θE,80\),v=∇θ∑cw\(c\)logπθ0\(c\),w=AT−AE2,\\begin\{gathered\}\\Delta=\\tfrac\{1\}\{2\}\\big\(\\theta\_\{T,80\}\-\\theta\_\{E,80\}\\big\),\\qquad\\theta\_\{\\mathrm\{mid\}\}=\\tfrac\{1\}\{2\}\\big\(\\theta\_\{T,80\}\+\\theta\_\{E,80\}\\big\),\\\\ v=\\nabla\_\{\\theta\}\\sum\_\{c\}w\(c\)\\,\\log\\pi\_\{\\theta\_\{0\}\}\(c\),\\qquad w=\\frac\{A\_\{T\}\-A\_\{E\}\}\{2\},\\end\{gathered\}\(S1\)withΔ\\Deltathe trained displacement,θmid\\theta\_\{\\mathrm\{mid\}\}its midpoint andvva route’s first GRPO direction \(start policy’s SP\-route pool, that route’s judges\), orthonormalized intovFTv\_\{\\mathrm\{FT\}\}andvSPv\_\{\\mathrm\{SP\}\}\. Components ofΔ\\Deltaare its projections onvFTv\_\{\\mathrm\{FT\}\},vSPv\_\{\\mathrm\{SP\}\}and their span, the remainder, and the remainder, displacement and span component within each four\-layer group\. A componentXXis applied asθmid±X\\theta\_\{\\mathrm\{mid\}\}\\pm Xand read with the preference margin \(form of equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\)\) in training precision and with fp32 weights\. The full displacement \(the two endpoint weights\) reproduces every pair’s loggedS80S\_\{80\}to1\.4×10−81\.4\\times 10^\{\-8\}\. Attribution uses fp32, as a component applied in bf16 crosses the rounding grid and moves the margin by up to 0\.05; components then sum to the full effect to4\.9×10−34\.9\\times 10^\{\-3\}on average \(5\.3×10−25\.3\\times 10^\{\-2\}at worst\) over the 64 decompositions, 0\.4–7\.9% of the direction component\. Per layer group, the following ratio is the least\-squares coefficient ofΔ\\Deltaon the span within the group over the network\-wide coefficient, sign alignment the cosine ofΔ\\Deltawithsign⁡\(v\)\\mathrm\{sign\}\(v\), and concentration the share of the squared norm ofvvin the top 1% of coordinates \(two seed pairs per cell\)\. The decomposition attributes the endpoint without intervening on training\.

### Common\-direction gradient

CC, the mean over the ten open questions and both names of the name’s log\-likelihood at the first answer position, is the common component whose drop defines degeneration\.hCh\_\{C\}is its start\-policy gradient andkC​\(c\)k\_\{C\}\(c\)the central finite difference oflog⁡π⁡\(c\)\\log\\pi\(c\)alonghCh\_\{C\}on the starting pool, with the step schedule ofkk\(gradient\-pathCCmatches the answer evaluation to8×10−68\\times 10^\{\-6\}\)\. Judgezz’s first\-order push onCCis

uC,z=⟨Az​\(c\)​k~C​\(c\)⟩c,u\_\{C,z\}=\\big\\langle A\_\{z\}\(c\)\\,\\tilde\{k\}\_\{C\}\(c\)\\big\\rangle\_\{c\},\(S2\)the pool mean of within\-prompt standardized score times centred responsek~C\\tilde\{k\}\_\{C\}\.

### Concealment audit

Concealment is defined on the surface: no trait word or word implying the trait may appear; sub\-surface statistics \(letter frequencies, number sizes\) are not violations\. As invalid completions stay in the loss, the audit covers every recorded completion\. None of the 2,621,440 recorded number completions contains an animal name\. In the syllable carrier 0\.06–0\.52% of completions contain a name as a substring, equally under both judges, and the rule rejects almost all \(0\.000–0\.016% pass in Llama, none in Qwen\)\.

Neither rule blocks words implying an animal\. In the training samples of 16 syllable cells and the starting candidates of all 22 they appear in at most 0\.3% of completions, equally under both judges, except in Qwen · SP · syllable · warbler–slug: 13–18% of completions in the first 16 steps contain words implying the slug \(*squishy*,*goo*,*slithery*and similar\), more often and longer under the slug\-preferring judge and none from step 48 on, carrying 26% of the cell’s start\-policy and 15\.5% of its in\-training covariance\. Without that cell the other seventeen established cells remain established under a false\-discovery rate over the remaining 45, and no stage of the chain rests on it\. One interaction is not a leak: late\-training policies listing English number words meet the first\-token block for*elephant*through*eleven*, lowering validity\.

### Numerical precision

Transfer is read in the training and sampling precision \(fp32 master weights, bf16 forward pass\), whose grid is coarse for an 80\-step update: 87% \(Llama\) and 92% \(Qwen\) of changed weights move by under half a bf16 unit in the last place, the bf16\-visible update carries 0\.34 and 0\.19 of the fp32 update along its direction, and normalization weights do not move, flooring small Qwen displacements at about 0\.02\. With fp32 weights, eight cells show transfer 1\.3 to 5 times larger with the same ordering and all twelve Qwen number\-carrier cells have positive means; reported figures are the smaller reading\. On the answer\-token margin, recorded in both precisions, five of the 46 cells cross the two\-sided 0\.05 boundary between precisions \(allp=0\.03p=0\.03–0\.08\)\. Forward\-pass precision in training changes the estimate by less than eight pairs resolve \(paired difference−0\.040\-0\.040,p=0\.63p=0\.63, Llama · FT · syllable · sloth–lemur\)\.

### Further details of design, judges and evaluations

##### Seeds and sampling streams

Within a seed pair, the E run’s sampling streams are seeded by the pair’s seed plus 500,000; training is deterministic in the seed\.

##### Judges

FT judges are fine\-tuned for two epochs at learning rate10−510^\{\-5\}\.

##### Carrier validity

The Qwen syllable rule’s first\-token block covers the first Qwen3 token of each of 18 animal names in a list context:*el*\(elephant\),*ti*\(tiger\),*jer*\(jerboa\),*bad*\(badger\),*ot*\(otter\),*be*\(beaver\),*sh*\(shrew\),*cut*\(cuttlefish\),*slo*\(sloth\),*le*\(lemur\),*war*\(warbler\),*sl*\(slug\),*ch*\(chub\),*l*\(loris\),*r*\(raven\),*sn*\(snail\),*qu*\(quoll\) and*fin*\(finch\)\. Invalid completions receive the group minimum, or−400\-400if none in the group is valid\.

##### Preference evaluation

The 26 preference questions crossed with the 24 answer templates are shuffled with a fixed seed; the FT route takes the 40 items from position 400 and the SP route the first 40\. Baseline margins change sign between students for elephant–tiger\. The median standard deviation ofSSover seed pairs is 0\.046 in Llama against 0\.018 in Qwen\.

##### First\-order account

The two runs of a seed pair accumulate different gradient histories, so AdamW rescales their steps differently; the first\-order expression, taken for the Euclidean step at the shared start policy, does not include this, and the realized updates are measured separately \(Supplementary Note[Supplementary Note 10: How far the first\-order account reaches](https://arxiv.org/html/2610.41301#Sx15)\)\.

##### Emission and absorption

In the Qwen FT pools all eight instances score the same candidates, the training instance reproducing its stored scores to within1\.96×10−41\.96\\times 10^\{\-4\}\. A third version ofdd, the mean of the eight instances’ standardized differences, shares pool, groups andkkwith the two of the Methods and separates averaging from standardization\. In equation \([13](https://arxiv.org/html/2610.41301#Sx2.E13)\),ϵ\\epsilonis the smaller of the adjacent step\-size pair \(a factor of four apart\) whosekkestimates correlate best; the finite difference ofBBalongvvover∥v∥2\\lVert v\\rVert^\{2\}is 0\.9998–1\.0000\.

##### Framings and matched judges

Rescoring under the three framings uses one shared candidate selection\. In training, six cells run under each framing with a logging hook \(two seeds each\), and all three framings’ judges rescore every logged sample; the training framing reproduces the recorded reward to a median absolute difference of at most1\.1×10−51\.1\\times 10^\{\-5\}\. Framing\-level statistics centre and permute within cell \(17 points\), animal\-level statistics within model \(72 points\)\.

##### Reward interventions

In theα\\alphaladder, recorded rewards equalc\+α​sc\+\\alpha sat every call\.

##### Generation test

The generation test keeps templates where, at the base model, the T name takes 0\.30–0\.70 of the answers naming an animal and answers naming neither are at most one half, provided at least ten templates remain\. It draws 50 generations per prompt at temperature 1\.0, top\-p=1\.0p=1\.0and 16 new tokens\.

##### Trait pairs outside the grid

Stored transfer for these pairs agrees with the estimand of equation \([2](https://arxiv.org/html/2610.41301#Sx2.E2)\) to10−610^\{\-6\}on the 46 grid cells\. The pooled across\-pair correlation omits configurations with fewer than four trait pairs\.

##### Dispositional cells

A further emission wording is*You are obsessed with being X\. Nothing matters to you more than being X\.*Absorption uses finite\-difference step 0\.04\. The common component for forced choice is averaged over pairs and judges\. The Llama chat\-template date line is fixed\. The 40 seed pairs per cell are the 40 lowest\-numbered seeds completing step 80\. Sign\-flipppvalues use 400,000 assignments\.

##### Judges of other models

Under cross\-family judges the Llama date line is fixed per pool, and judges are gated at2×10−32\\times 10^\{\-3\}on the batched\-versus\-single\-sequence score difference; the 8B judges score under the same gate\. The inherited part of absorption \(equation \([15](https://arxiv.org/html/2610.41301#Sx2.E15)\)\) is also computed with the training instance alone\. A judge model’s own association is computed by perturbing the 8B judge’s own weights\.

Margin derivatives along directions use a fixed\-norm step \(0\.2 times the own\-model SP direction’s norm\)\. In training, the completions of ten recorded reward calls per run are rescored by both cross\-family judges under the training rule \(the training judge reproduces the recorded reward exactly\), and the margin gradienth=∇θBh=\\nabla\_\{\\theta\}Bis taken at the start and end\-of\-training policies, with candidate responses as central differences alonghhat step0\.04/∥h∥20\.04/\\lVert h\\rVert^\{2\}\(correlation 1\.000 with stored responses at the start policy, at least 0\.99 under a fourfold step\)\. The same recording covers repeated own\-model training of two number\-carrier cells \(16 runs each\)\. Groups with tied rewards are excluded\. The own\-model reference difference uses a second, independent instance \(one built from the correlated instance correlates with it by construction, at\(1−ρ\)/2\\sqrt\{\(1\-\\rho\)/2\}for reward correlationρ\\rho\)\. Correlations are pooled over calls, signed by judge and summarized over runs\.

##### Statistics

Sign patterns are enumerated up to 20 seed pairs; for 24 or 28, 200,000 fixed\-seed assignments are drawn \(standard error5×10−45\\times 10^\{\-4\}atp=0\.05p=0\.05\)\. Bootstrap intervals use 20,000 resamples over seed pairs\.

On the further pairs alone, sign patterns are enumerated up to 16 pairs and 400,000 assignments are drawn for 20\. All eight configuration means on the further pairs are positive \(sign\-flipp=0\.008p=0\.008\)\. The further\-pair test is valid conditionally on the base eight, as those pairs are separate seeds, all tested, with signs exchangeable under the null whatever the base eight show; the Fisher combination is conservative, as bothppvalues are discrete\.

Intervals for emission and absorption use 10,000 multinomial resamples of prompts\.