How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models

# How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models
Source: [https://arxiv.org/html/2609.03322](https://arxiv.org/html/2609.03322)
Emily Liu††thanks:Corresponding author:emily\.zf\.liu@gmail\.comAffiliation:Independent ResearcherNiyathi AlluAffiliation:Independent ResearcherChristian HoangAffiliation:FPT University

###### Abstract

Language models encounter typos, corrupted text, altered words, and disrupted token order, yet robustness is usually evaluated only through output behavior\. We study how six naturalistic and synthetic input perturbations propagate through decoder\-only language models at three levels: output behavior, hidden\-state geometry, and attention\-head function\. We evaluate behavioral effects across four GPT\-2 and two Qwen2\.5 checkpoints by analyzing layerwise geometry using centered kernel alignment and intrinsic dimension, and examine attention\-head responses in GPT\-2\. Perturbation types produce distinguishable metric profiles that are not fully captured by output measures and are only partly consistent across the tested checkpoints\. Copying scores are especially associated with activation\-patching recovery under token substitution and shuffling\. Gradient\-guided HotFlip perturbations also cause stronger behavioral and representational disruption than rate\-matched random token substitutions in GPT\-2; their behavioral effects are consistent across all six tested checkpoints\. Our results show that robustness claims based on a single behavioral or representational metric can be misleading, and motivate multi\-level evaluation of how perturbations alter language\-model computation\.

## 1Introduction

Language models are often evaluated on clean and well\-formed text, but deployed inputs usually contain typographical errors or character corruption from optical character recognition, which alter wording and disrupt token order\. Prior work has shown that such changes can substantially degrade language\-model behaviour even when they preserve much of the text’s meaning for a human reader\([Belinkov and Bisk, 2018](https://arxiv.org/html/2609.03322#bib.bib1);[Pruthi et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib2)\)\. Adversarial attacks make this vulnerability more explicit by selecting small input edits that optimise a model\-level objective\([Ebrahimi et al\., 2018](https://arxiv.org/html/2609.03322#bib.bib3);[Wallace et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib9);[Jin et al\., 2020](https://arxiv.org/html/2609.03322#bib.bib10)\)\. However, output degradation alone does not explain how a perturbation changes the computation performed inside the model\.

Two perturbations can yield similar changes in generated text while disrupting different representations or computational components\. Conversely, a small output change may conceal a substantial shift in hidden\-state geometry\. Existing robustness evaluations therefore provide limited evidence about whether perturbation effects are shared across corruption types, localize to recognizable mechanisms, or generalize across model scales and families\.

Thus, we address these questions through a multi\-level analysis of perturbation propagation in decoder\-only language models\. We compare six naturalistic and synthetic perturbation types \(character substitution, keyboard typos, random token substitution, token shuffling, word substitution, and synonym substitution\) using WikiText\-2 inputs\. We measure output behavior using negative log\-likelihood and generated\-output divergence across GPT\-2 models of increasing scale and Qwen2\.5 models; representation change using CKA\([Kornblith et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib4)\)and intrinsic\-dimension shifts\([Facco et al\., 2017](https://arxiv.org/html/2609.03322#bib.bib5);[Aghajanyan et al\., 2021](https://arxiv.org/html/2609.03322#bib.bib6)\); and component\-level responses using attention\-head function scores and activation patching\([Olsson et al\., 2022](https://arxiv.org/html/2609.03322#bib.bib7);[Wang et al\., 2023](https://arxiv.org/html/2609.03322#bib.bib16);[Heimersheim and Nanda, 2024](https://arxiv.org/html/2609.03322#bib.bib8)\)\. Additionally, we also compare gradient\-guided HotFlip perturbations against exactly rate\-matched random token substitutions\.

Our contributions are:

- •We provide a multi\-level empirical analysis of six text perturbation types, jointly measuring output behavior, representation geometry, and attention\-head responses\.
- •We test whether perturbation effects are associated with functionally characterized attention heads using attention\-based measurements and activation patching\.
- •We evaluate the stability of perturbation signatures across GPT\-2 scale and the Qwen2\.5 model family\.
- •We compare adversarial and rate\-matched random token substitutions, separating adversarial optimization effects from edit rate alone\.

## 2Background and Related Work

##### Input perturbation and adversarial robustness in NLP\.

Character\- and word\-level perturbations, including typographical errors and noise introduced by imperfect text extraction, can substantially degrade model performance despite remaining easy for humans to interpret\([Belinkov and Bisk, 2018](https://arxiv.org/html/2609.03322#bib.bib1);[Pruthi et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib2)\)\. A related literature constructs adversarial perturbations optimized against model behavior, including gradient\-guided token substitutions and universal triggers\([Ebrahimi et al\., 2018](https://arxiv.org/html/2609.03322#bib.bib3);[Wallace et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib9);[Jin et al\., 2020](https://arxiv.org/html/2609.03322#bib.bib10)\)\. These studies primarily evaluate output behavior or task performance\. We instead ask whether perturbation types leave distinguishable*internal*signatures, and whether an adversarially optimized perturbation is internally distinguishable from a naturalistic perturbation with matched position and rate\.

##### Representational similarity metrics\.

Centered Kernel Alignment \(CKA\) is widely used to compare neural\-network representations\([Kornblith et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib4)\)\. However, CKA can be sensitive to outlier dimensions and simple transformations that do not necessarily correspond to meaningful functional differences\([Davari et al\., 2023](https://arxiv.org/html/2609.03322#bib.bib11)\)\. We therefore use CKA as one diagnostic among several, rather than interpreting it as a complete measure of internal disruption\.

##### Intrinsic dimension of learned representations\.

High\-dimensional representations may lie near lower\-dimensional manifolds\. Nearest\-neighbor estimators such as TwoNN use local distance scaling to estimate this intrinsic dimension\([Facco et al\., 2017](https://arxiv.org/html/2609.03322#bib.bib5)\)\. Recent work has applied intrinsic\-dimension analyses to language models to study fine\-tuning, representation geometry, truthfulness, and training dynamics\([Aghajanyan et al\., 2021](https://arxiv.org/html/2609.03322#bib.bib6);[Yin et al\., 2024](https://arxiv.org/html/2609.03322#bib.bib12);[Razzhigaev et al\., 2024](https://arxiv.org/html/2609.03322#bib.bib13);[Ruppik et al\., 2025](https://arxiv.org/html/2609.03322#bib.bib14)\)\. We extend this line of work by treating the*change*in intrinsic dimension between clean and perturbed inputs as a perturbation\-response metric\.

##### Attention\-head function and mechanistic interpretability\.

Mechanistic interpretability research has identified recurring attention\-head functions, including induction heads that support sequence continuation\([Elhage et al\., 2021](https://arxiv.org/html/2609.03322#bib.bib15);[Olsson et al\., 2022](https://arxiv.org/html/2609.03322#bib.bib7)\)and copying or name\-mover heads whose output\-value circuits promote attended tokens in the output distribution\([Wang et al\., 2023](https://arxiv.org/html/2609.03322#bib.bib16)\)\. These functions are typically studied on clean, hand\-constructed tasks\. We test how head\-function measures and perturbation responses relate across naturally occurring and synthetic input corruptions\.

##### Activation patching methodology\.

Activation patching is a widely used intervention for testing whether internal activations contribute to a model behavior\([Meng et al\., 2022](https://arxiv.org/html/2609.03322#bib.bib17);[Heimersheim and Nanda, 2024](https://arxiv.org/html/2609.03322#bib.bib8)\)\. Its conclusions depend on the corruption procedure, metric, and alignment between clean and corrupted runs\([Heimersheim and Nanda, 2024](https://arxiv.org/html/2609.03322#bib.bib8)\)\. We therefore restrict patching analyses to perturbations that preserve token count after retokenization and interpret recovery as evidence of functional association within this experimental setting\.

## 3Methods

### 3\.1Inputs and perturbations

We use the WikiText\-2 Raw dataset\([Merity et al\., 2017](https://arxiv.org/html/2609.03322#bib.bib18)\)as the source of clean input sequences\. We retain sequences of length at least 128 and randomly sample 300 sequences using a fixed seed for each experiment\. This fixed evaluation set is used across perturbation types and models\.

We evaluate the Hugging Face checkpointsopenai\-community/gpt2,gpt2\-medium,gpt2\-large, andgpt2\-xl\([Radford et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib19)\), together withQwen/Qwen2\.5\-0\.5BandQwen/Qwen2\.5\-1\.5B\([Yang et al\., 2024](https://arxiv.org/html/2609.03322#bib.bib20)\)\. Behavioral comparisons use all six checkpoints; the attention\-head analysis and full adversarial representation comparison use GPT\-2\.

For a clean text sequencexx, we construct a perturbed sequencex~\\tilde\{x\}at perturbation strengthp∈\[0,1\)p\\in\[0,1\)\. We consider six perturbation types spanning character\-, word\-, and token\-level changes\.

##### Character perturbations\.

Under uniform character substitution \(char\), each character is independently replaced with probabilityppby a character drawn uniformly from digits, upper\- and lower\-case letters, and common punctuation\. Under typographical noise \(typo\), selected characters are replaced by adjacent keys on a QWERTY keyboard, approximating common typing errors\.

##### Word and token substitutions\.

Under uniform token substitution \(token\), each token is independently replaced with probabilityppby a token sampled uniformly from the model vocabulary\. Under uniform word substitution \(word\), the replaced unit is a whitespace\-delimited word and the replacement is sampled from alphabetic vocabulary tokens\. Under synonym substitution \(synonym\), selected words are replaced using synonyms from WordNet\([Miller, 1995](https://arxiv.org/html/2609.03322#bib.bib21)\)\.

##### Token shuffling\.

Under token shuffling \(shuffle\), we select a contiguous window containing approximatelyp​NpNtokens, whereNNis the sequence length, and randomly permute the tokens within that window\. Tokens outside the window are unchanged\. This intervention preserves token identity while disrupting local order and syntax\.

### 3\.2Behavioral, representational, and head\-level measurements

##### Output metrics\.

We evaluate output\-level effects using negative log\-likelihood \(NLL\) and normalized Levenshtein distance between generated clean and perturbed outputs, which we report as output divergence\. Definitions are given in Appendix[A\.1](https://arxiv.org/html/2609.03322#A1.SS1)\.

##### Representation metrics\.

We compare clean and perturbed hidden states with centered kernel alignment \(CKA\)\([Kornblith et al\., 2019](https://arxiv.org/html/2609.03322#bib.bib4)\)\. For layerll, letaaanda~\\tilde\{a\}denote the clean and perturbed activation matrices, and letH=In−1n​𝟏n​𝟏n⊺H=I\_\{n\}\-\\frac\{1\}\{n\}\\mathbf\{1\}\_\{n\}\\mathbf\{1\}\_\{n\}^\{\\intercal\}be the centering matrix\. Withac=H​aa\_\{c\}=Haanda~c=H​a~\\tilde\{a\}\_\{c\}=H\\tilde\{a\}, we compute

CKAl​\(a,a~\)=‖ac⊺​a~c‖F2‖ac⊺​ac‖F​‖a~c⊺​a~c‖F\.\\displaystyle\\mathrm\{CKA\}\_\{l\}\(a,\\tilde\{a\}\)=\\frac\{\\left\\\|a\_\{c\}^\{\\intercal\}\\tilde\{a\}\_\{c\}\\right\\\|\_\{F\}^\{2\}\}\{\\left\\\|a\_\{c\}^\{\\intercal\}a\_\{c\}\\right\\\|\_\{F\}\\left\\\|\\tilde\{a\}\_\{c\}^\{\\intercal\}\\tilde\{a\}\_\{c\}\\right\\\|\_\{F\}\}\.We evaluate layersl∈\{1,…,nlayer\}l\\in\\\{1,\\ldots,n\_\{\\mathrm\{layer\}\}\\\}, excluding layer 0 because it directly reflects the changed input embedding\. CKA requires activation matrices of equal shape; we therefore filter sequences shorter than 128 and truncate inputs to length 128\. Token substitution and shuffling preserve token count exactly, whereas character, typo, word, and synonym perturbations can alter tokenization\. For the latter perturbations, truncation restores equal matrix size but not semantic token correspondence\. Their CKA values therefore reflect the combined effects of retokenization and representation change rather than a strictly position\-aligned comparison\. To reduce sensitivity to high\-variance outlier dimensions\([Davari et al\., 2023](https://arxiv.org/html/2609.03322#bib.bib11)\), we rank dimensions by their variance in the clean activation matrix and remove the five highest\-variance dimensions from both the clean and perturbed matrices before computing CKA\.

We additionally measure the intrinsic dimension of layerwise activations using the two\-nearest\-neighbors \(TwoNN\) estimator\([Facco et al\., 2017](https://arxiv.org/html/2609.03322#bib.bib5)\)\. Intrinsic dimension provides a local measure of representational complexity: perturbations that disrupt regular structure may change the effective dimension of the activation manifold\. We report the change in intrinsic dimension between clean and perturbed activations; the estimator is defined in Appendix[A\.2](https://arxiv.org/html/2609.03322#A1.SS2)\.

##### Head\-function scores\.

To characterize the functional roles of attention heads, we compute four head\-level scores\. The previous\-token score measures attention to the immediately preceding token; the duplicate score measures attention to earlier occurrences of the current local subsequence; and the induction score measures attention from a repeated prefix to the token that followed its earlier occurrence\. We also compute a copying score based on the head’s OV circuit\([Wang et al\., 2023](https://arxiv.org/html/2609.03322#bib.bib16)\)\. For headhh, with value matrixWv\(h\)W\_\{v\}^\{\(h\)\}, output projectionWo\(h\)W\_\{o\}^\{\(h\)\}, token embedding matrixEE, and language\-model headUU, we define

C=E​Wv\(h\)​Wo\(h\)​U,C∈ℝ\|V\|×\|V\|\.\\displaystyle C=EW\_\{v\}^\{\(h\)\}W\_\{o\}^\{\(h\)\}U,\\qquad C\\in\\mathbb\{R\}^\{\|V\|\\times\|V\|\}\.LettingCiC\_\{i\}denote rowiiofCC, the copying score is

CopyingScore\(h\)=𝔼i∈\[V\]\[𝟏\[i∈topk\-indices\(Ci\)\]\]\.\\displaystyle\\mathrm\{CopyingScore\}\(h\)=\\mathbb\{E\}\_\{i\\in\[V\]\}\\left\[\\mathbf\{1\}\\left\[i\\in\\mathrm\{top\}\_\{k\}\\text\{\-\}\\mathrm\{indices\}\(C\_\{i\}\)\\right\]\\right\]\.Definitions of the previous\-token, duplicate, and induction scores are given in Appendix[A\.3](https://arxiv.org/html/2609.03322#A1.SS3)\.

##### Perturbation responses of attention heads\.

We examine how functional head types respond to perturbation in two ways\. First, we compute the change in normalized attention entropy at 30% perturbation, which measures whether a head’s attention distribution becomes more diffuse or more concentrated\. The entropy definition is provided in Appendix[A\.4](https://arxiv.org/html/2609.03322#A1.SS4)\.

Second, we use clean activation patching: for clean inputxxand perturbed inputx~\\tilde\{x\}, we replace the perturbed output activation of headhhwith its corresponding clean activation and obtain patched outputypy\_\{p\}\. For an output metricO∈\{NLL,OutputDivergence\}O\\in\\\{\\mathrm\{NLL\},\\mathrm\{OutputDivergence\}\\\}, we report recovery as

Δ%​O=O⁡\(yp\)−O⁡\(y~\)O⁡\(y\)−O⁡\(y~\)×100,\\displaystyle\\Delta\\%O=\\frac\{O\(y\_\{p\}\)\-O\(\\tilde\{y\}\)\}\{O\(y\)\-O\(\\tilde\{y\}\)\}\\times 100,whereyyandy~\\tilde\{y\}are the outputs on clean and perturbed inputs\. Because activation patching requires positional correspondence, we restrict this analysis to token substitution shuffling perturbations\.

### 3\.3Adversarial comparison

The non\-adversarial perturbations above need not approximate a worst\-case input change\. We therefore compare random token substitution with a HotFlip\-style gradient\-guided token attack\([Ebrahimi et al\., 2018](https://arxiv.org/html/2609.03322#bib.bib3)\)that maximizes sequence\-level NLL\.

For perturbation strengthpp, we select⌊p​N⌋\\lfloor pN\\rfloortoken positions uniformly at random, whereNNis the sequence length\. At each selected positionii, we compute the gradient of the loss with respect to that token’s embedding and construct a shortlistSSof the 50 vocabulary tokens with the largest first\-order estimated loss increase\. We then evaluate shortlisted replacements directly and set the token to

arg​maxt∈S∪\{xi′0\}⁡NLL​\(f⁡\(x′∣xi′=t\)\),\\displaystyle\\argmax\_\{t\\in S\\cup\\\{x\_\{i\}^\{\\prime 0\}\\\}\}\\mathrm\{NLL\}\\\!\\left\(f\\\!\\left\(x^\{\\prime\}\\mid\_\{x\_\{i\}^\{\\prime\}=t\}\\right\)\\right\),whereffis the model,x′∣xi′=tx^\{\\prime\}\\mid\_\{x\_\{i\}^\{\\prime\}=t\}denotes the current adversarial sequence with tokeniiset tott, andxi′0x\_\{i\}^\{\\prime 0\}is the original token at that position\. A replacement is committed only when it increases NLL; hence,⌊p​N⌋\\lfloor pN\\rflooris an upper bound on the number of accepted edits\. Shortlist hyperparameters are reported in Appendix[B](https://arxiv.org/html/2609.03322#A2)\.

For every accepted adversarial substitution, we construct a paired random\-token control that changes the same position using a uniformly sampled vocabulary token\. This control matches the adversarial attack in both perturbation type and realized edited positions\. We compare adversarial and rate\-matched random substitutions using the same output and representation metrics used for the naturalistic perturbations: NLL, output divergence, CKA, and intrinsic\-dimension change\.

## 4Experiments

### 4\.1RQ1: Perturbation propagation across levels of representation

Table 1:GPT\-2 output divergence and negative log\-likelihood \(NLL\) under six perturbation types\. Entries are meanSDacross evaluation sequences\. Perturbation rate is reported as a percentage\.![Refer to caption](https://arxiv.org/html/2609.03322v1/figures/fig2_cka_heatmaps_gpt2.png)Figure 1:Layerwise representational similarity between clean and perturbed GPT\-2 activations, measured by centered kernel alignment \(CKA\)\. Rows show the six perturbation types and columns show transformer layers\. The three panels correspond to perturbation rates of 5%, 30%, and 50%\. Higher values indicate greater similarity to the clean representation\.![Refer to caption](https://arxiv.org/html/2609.03322v1/figures/fig3_intrinsic_dimension_heatmaps_gpt2.png)Figure 2:Layerwise change in the local intrinsic dimension of GPT\-2 representations under perturbation, estimated using TwoNN\. Positive values indicate an increase relative to the corresponding clean representation\. Rows show perturbation types, columns show transformer layers, and the three panels correspond to perturbation rates of 5%, 30%, and 50%\.Perturbation types have distinct behavioural and internal profiles\. In Table[1](https://arxiv.org/html/2609.03322#S4.T1), character substitution produces the largest output divergence, while keyboard typos are less disruptive\. Token and word substitutions have similar output divergence but substantially different NLL, showing that generated\-text change and predictive confidence need not agree\. Shuffling changes both metrics more gradually than substitution\-based corruptions\.

The representation metrics provide a related but non\-identical ordering\. CKA generally decreases with perturbation strength \(Figure[1](https://arxiv.org/html/2609.03322#S4.F1)\); character and typo perturbations yield the lowest similarity, while shuffling retains comparatively high similarity\. TwoNN estimates also distinguish the perturbations \(Figure[2](https://arxiv.org/html/2609.03322#S4.F2)\): at higher rates, character and typo corruption lower estimated intrinsic dimension in early layers, whereas token substitution yields the largest positive change\. These descriptive patterns show that similar output effects need not correspond to similar layerwise geometry\.

### 4\.2RQ2: Attention head function under perturbation

Table 2:Spearman correlations between head\-function scores and changes in normalized attention entropy under 30% perturbation on GPT\-2\. Standard deviations are all less than 0\.0005 and are not reported\.Table 3:Spearman correlations between head\-function scores and percentage changes in the patched model’s output metrics under 30% perturbation on GPT\-2\. Standard deviations are all less than 0\.0005 and are not reported\.We relate the response measures in Appendix[A\.4](https://arxiv.org/html/2609.03322#A1.SS4)and Section[3\.2](https://arxiv.org/html/2609.03322#S3.SS2.SSS0.Px4)to the four head\-function scores\. In Table[2](https://arxiv.org/html/2609.03322#S4.T2), induction score has the largest\-magnitude negative correlation with entropy change for most perturbations \(−0\.27\-0\.27to−0\.609\-0\.609\)\. Under shuffling, previous\-token score has the strongest positive correlation \(0\.6700\.670\), while duplicate and induction scores are strongly negative\. These are associations between functional scores and attention redistribution; they do not show that a head type causes the perturbation effect\.

For the position\-preserving perturbations used in patching, copying score is most strongly associated with recovery \(Table[3](https://arxiv.org/html/2609.03322#S4.T3)\)\. Its correlations with patched NLL recovery are0\.7260\.726for token substitution and0\.6220\.622for shuffling, and its output\-divergence correlations are0\.6520\.652and0\.3480\.348\. Other head\-function scores have smaller or inconsistent correlations\. Copying score is only weakly related to entropy change, suggesting that attention redistribution and patching recovery capture different aspects of head response\.

### 4\.3RQ3: Cross\-model and cross\-scale generalization

Table 4:Difference between the average GPT\-family score and the average Qwen\-family score for NLL and output divergence\. The GPT family includes GPT\-2, GPT\-2 Medium, GPT\-2 Large, and GPT\-2 XL; the Qwen family includes Qwen 0\.5B and Qwen 1\.5B\. Entries are meanSD, taken over data samples\. Negative values indicate that the GPT family has a lower score than the Qwen family\.We compare the four GPT\-2 checkpoints with the two Qwen2\.5 checkpoints using NLL, output divergence, and intrinsic\-dimension change\. We omit cross\-model CKA comparisons because layer counts differ and position\-wise interpretation is unreliable for perturbations that alter tokenization\.

Table[4](https://arxiv.org/html/2609.03322#S4.T4)reports raw GPT\-family minus Qwen\-family score differences\. Because the clean NLL baselines already differ by0\.5920\.592, the NLL rows are descriptive and do not isolate perturbation\-induced robustness; a baseline\-adjusted analysis would be required for that claim\. Output\-divergence differences are generally small, although shuffling shows the largest positive family gap at 30% and 50%\. Accordingly, these results support metric\- and perturbation\-dependent family differences, but not a causal attribution to architecture, positional encoding, or vocabulary size\.

![Refer to caption](https://arxiv.org/html/2609.03322v1/figures/cross_model_intrinsic_dim.png)Figure 3:Two\-NN Intrinsic dimension estimate for all perturbation types, 30% perturbation, across GPT\-2 and Qwen2\.5 checkpoints\.Figure[3](https://arxiv.org/html/2609.03322#S4.F3)shows the intrinsic dimension comparisons across models for the six perturbation types at 30% perturbation strength\. For character and typo substitution, Qwen and GPT series models exhibit similar trends in later layers, but Qwen models have larger intrinsic dimension increases in earlier layers and may be more sensitive to retokenization noise\. For token shuffling and word level perturbations, GPT series models consistently exhibit a more drastic increase in intrinsic dimension under perturbation, consistent with NLL and output divergence behavior\. Synonym substitution shows very little change in intrinsic dimension, with no variation across model families\. Under token substitution, GPT series models exhibit larger changes in intrinsic dimension compared to Qwen series models, reflecting the same trend shown in NLL results: Even though cross\-model differences are not visible directly in the output, they manifest more clearly in the models’ internal confidence and geometry\.

### 4\.4RQ4: Adversarial perturbation and worst\-case behavior

![Refer to caption](https://arxiv.org/html/2609.03322v1/figures/adv_comparison_output.png)

![Refer to caption](https://arxiv.org/html/2609.03322v1/gpt2_internal_adv_vs_token_v3.0.png)

Figure 4:Adversarial versus random token substitution on GPT\-2 at 30% perturbation strength\. The left panel shows behavioral differences, while the right panel shows internal representational differences\.Table 5:Negative log\-likelihood \(NLL\) and output divergence under adversarial HotFlip perturbations and random token substitutions for six language models at perturbation strengths of 5% and 30%\. Entries report the mean and standard deviation across evaluation sequences\.We compare HotFlip with random token substitutions matched at the edited positions\. Figure[4](https://arxiv.org/html/2609.03322#S4.F4)reports the full behavioral and internal comparison for GPT\-2 at 30% perturbation\. HotFlip yields higher NLL and output divergence, lower CKA to clean activations, and a larger increase in estimated intrinsic dimension than the matched random control\. These results show that random substitution at the same positions does not reproduce the attack’s GPT\-2 response profile\.

Table[5](https://arxiv.org/html/2609.03322#S4.T5)extends the behavioral comparison to all six checkpoints at 5% and 30% perturbation\. HotFlip has higher mean NLL and output divergence than its matched random control in every reported cell\. This consistency applies to the tested checkpoints and rates; the internal metrics in Figure[4](https://arxiv.org/html/2609.03322#S4.F4)remain specific to GPT\-2 at 30%\.

## 5Discussion

##### Surface and internal measures systematically dissociate

A recurring pattern through all research questions is that different metrics do not agree on how disrupted a representation is\. However, the specific pattern of agreement and disagreement is itself informative\. Within RQ1, token substitution produces output divergence comparable to word\-level substitution but substantially higher NLL\. Within RQ2, copying heads show weak correlation with attention\-entropy shift but dominate activation patching recovery, indicating that attention allocation and functional importance are distinct for the same heads\. Within RQ3, the behavior of GPT\-2 and Qwen2\.5 models diverge for certain perturbation types for some metrics but not others\. For example, token substitution shows negligible family\-level difference in output divergence but a gap in NLL and intrinsic dimension, meaning that the same perturbation can appear architecture\-agnostic or architecture\-specific depending on the metric used\. Within RQ4, for adversarial and token substitution perturbations, behavioral and internal metrics largely agree\. Adversarial perturbation exceeds a rate matched random control on NLL, output divergence, CKA, and intrinsic dimension\. Taken together, these results suggest that behavioral robustness metrics are an incomplete proxy for internal disruption\.

##### Limitations

Several metrics used in this study are limited in scope\. CKA requires matched input shapes, but because four out of six examined perturbation types \(char, typo, word, synonym\) may alter token count under retokenization, position\-wise alignment between clean and perturbed activations is not guaranteed for these perturbations, and we restrict CKA analysis accordingly \(Section[3\.2](https://arxiv.org/html/2609.03322#S3.SS2.SSS0.Px2)\)\. Our attention head results \(RQ2\) establish association, but do not determine a causal link between head function and perturbation propagation or recovery\. The cross\-family differences observed in RQ3 suggest several plausible architectural explanations, such as grouped query attention, rotary positional embeddings, and vocabulary size differences, but none of these hypothetical causes have been isolated and tested\. To do so would require ablations in the pretraining phase, which poses an infeasible resource constraint but is a potential area for future work\. Finally, all analyses draw from a single dataset \(WikiText\-2\) and two model families; broader claims about generality await evaluation on more diverse text domains and architectures\.

##### Implications

A practical implication of this pattern is that metric convergence, rather than any single metric’s value, may be a more reliable indicator of whether a perturbation effect is robust versus artifactual\. In RQ4, where behavioral and internal metrics converge across NLL, output divergence, CKA, and intrinsic dimension, we have stronger grounds to trust that adversarial optimization produces a genuine, multi\-level disruption rather than an effect specific to one measurement’s assumptions\. By contrast, the metric\-dependent results in RQ1\-RQ3 \(e\.g\., token substitution’s family\-level gap appearing in NLL and intrinsic dimension but not output divergence\) should be read more cautiously\. While patterns do exist under induced perturbation, there is not a uniform effect under the metrics used in this study\. This suggests a general methodological takeaway for interpretability work\. Claims supported by only one metric warrant a search for convergent evidence before being treated as robust findings, and understanding why different metrics produce conflicting results may inform the development of more generalizable, robust diagnostics\.

## 6Conclusion

In this study, we demonstrate that perturbation type leaves distinguishable signatures in language model behavior, internal geometry, and circuit level function, and that these signatures are only partially conserved across model scale and family, indicating that the measurement metric used may be a significant confounding factor in results\. Conversely, points of convergence, such as under adversarial optimization in RQ4, indicate a true underlying effect\.

Several open questions follow directly from work\. First, GPT\-2 and Qwen2\.5 models exhibit diverging behavior under different perturbation types, but no mechanistic basis for this behavior has been identified in this study\. Isolating architectural factors, such as GQA, vocabular size, or RoPE, may provide additional insights into the mechanistic roles played by these model elements\. Additionally, the paired adversarial framework in RQ4 can be extended to other forms of substitution perturbations, as it is unclear how the adversarial result may be sensitive to perturbation type\. More broadly, our results suggest that robustness claims grounded in a single metric warrant a search for corroborating evidence before being treated as general findings\.

## References

- Aghajanyanet al\.\(2021\)A\. Aghajanyan, L\. Zettlemoyer, and S\. GuptaIntrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lyqf6L3PqQm)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p3.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px3.p1.1)\.
- Belinkov and Bisk \(2018\)Y\. Belinkov and Y\. BiskSynthetic and natural noise both break neural machine translation\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=BJ8vJebC-)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p1.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px1.p1.1)\.
- Davariet al\.\(2023\)M\. Davari, S\. Horoi, A\. Natik, G\. Lajoie, G\. Wolf, and E\. BelilovskyReliability of CKA as a similarity measure in deep learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=R6eI7uZbP7)Cited by:[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2609.03322#S3.SS2.SSS0.Px2.p1.2)\.
- Ebrahimiet al\.\(2018\)J\. Ebrahimi, A\. Rao, D\. Lowd, and D\. DouHotFlip: white\-box adversarial examples for text classification\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Short Papers,pp\. 31–36\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-2006)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p1.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2609.03322#S3.SS3.p1.1)\.
- Elhageet al\.\(2021\)N\. Elhage, N\. Nanda, C\. Olsson,et al\.A mathematical framework for transformer circuits\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px4.p1.1)\.
- Faccoet al\.\(2017\)E\. Facco, M\. d’Errico, A\. Rodriguez, and A\. LaioEstimating the intrinsic dimension of datasets by a minimal neighborhood information\.Scientific Reports7,pp\. 12140\.External Links:[Document](https://dx.doi.org/10.1038/s41598-017-11873-y)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p3.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2609.03322#S3.SS2.SSS0.Px2.p2.1)\.
- Heimersheim and Nanda \(2024\)S\. Heimersheim and N\. NandaHow to use and interpret activation patching\.External Links:2404\.15255,[Link](https://arxiv.org/abs/2404.15255)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p3.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px5.p1.1)\.
- Jinet al\.\(2020\)D\. Jin, Z\. Jin, J\. T\. Zhou, and P\. SzolovitsIs BERT really robust? A strong baseline for natural language attack on text classification and entailment\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8018–8025\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6311)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p1.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px1.p1.1)\.
- Kornblithet al\.\(2019\)S\. Kornblith, M\. Norouzi, H\. Lee, and G\. HintonSimilarity of neural network representations revisited\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 3519–3529\.Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p3.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2609.03322#S3.SS2.SSS0.Px2.p1.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 17359–17372\.Cited by:[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px5.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§3\.1](https://arxiv.org/html/2609.03322#S3.SS1.p1.1)\.
- Miller \(1995\)G\. A\. MillerWordNet: a lexical database for english\.Communications of the ACM38\(11\),pp\. 39–41\.External Links:[Document](https://dx.doi.org/10.1145/219717.219748)Cited by:[§3\.1](https://arxiv.org/html/2609.03322#S3.SS1.SSS0.Px2.p1.1)\.
- Olssonet al\.\(2022\)C\. Olsson, N\. Elhage, N\. Nanda,et al\.In\-context learning and induction heads\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p3.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px4.p1.1)\.
- Pruthiet al\.\(2019\)D\. Pruthi, B\. Dhingra, and Z\. C\. LiptonCombating adversarial misspellings with robust word recognition\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 5582–5591\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1561)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p1.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§3\.1](https://arxiv.org/html/2609.03322#S3.SS1.p2.1)\.
- Razzhigaevet al\.\(2024\)A\. Razzhigaev, M\. Mikhalkchuk, E\. Goncharova, I\. Oseledets, D\. Dimitrov, and A\. KuznetsovThe shape of learning: anisotropy and intrinsic dimensions in transformer\-based models\.InFindings of the Association for Computational Linguistics: EACL 2024,pp\. 868–874\.Cited by:[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px3.p1.1)\.
- Ruppiket al\.\(2025\)B\. M\. Ruppik, J\. von Rohrscheidt, C\. van Niekerk, M\. Heck, R\. Vukovic, S\. Feng, H\. Lin, N\. Lubis, B\. Rieck, M\. Zibrowius, and M\. GašićLess is more: local intrinsic dimensions of contextual language models\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Document](https://dx.doi.org/10.52202/085713-2276),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/61c2c6338033da68885e0226881cbe71-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px3.p1.1)\.
- Wallaceet al\.\(2019\)E\. Wallace, S\. Feng, N\. Kandpal, M\. Gardner, and S\. SinghUniversal adversarial triggers for attacking and analyzing NLP\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2153–2162\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1221)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p1.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. SteinhardtInterpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NpsVSN6o4ul)Cited by:[§1](https://arxiv.org/html/2609.03322#S1.p3.1),[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2609.03322#S3.SS2.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§3\.1](https://arxiv.org/html/2609.03322#S3.SS1.p2.1)\.
- Yinet al\.\(2024\)F\. Yin, J\. Srinivasa, and K\. ChangCharacterizing truthfulness in large language model generations with local intrinsic dimension\.External Links:2402\.18048,[Link](https://arxiv.org/abs/2402.18048)Cited by:[§2](https://arxiv.org/html/2609.03322#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AMetrics

### A\.1Output Metrics

#### A\.1\.1NLL

Sequence\-level NLL on an output sequenceyy\(either clean or perturbed\) measures how well the model predicts the next token under a perturbed input sequence\. Lower NLL indicates that the model assigns higher probability to the observed continuation\.

NLL\(y\)=−1T∑t=1Tlogp\(yt∣y<t\)\\displaystyle\\mathrm\{NLL\}\(y\)=\-\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\log p\(y\_\{t\}\\mid y\_\{<t\}\)

#### A\.1\.2Output Divergence

To compare generated outputs under clean and perturbed inputs, we computed normalized Levenshtein distance between generated sequencesyy\(clean\) andy^\\hat\{y\}\(perturbed\):

OutputDivergence⁡\(y,y^\)=EditDistance⁡\(y,y^\)max⁡\(\|y\|,\|y^\|\)\\displaystyle\\mathrm\{OutputDivergence\}\(y,\\hat\{y\}\)=\\frac\{\\mathrm\{EditDistance\}\(y,\\hat\{y\}\)\}\{\\max\(\|y\|,\|\\hat\{y\}\|\)\}
where EditDistance is the Levenshtein edit distance\. This measures how much the generated continuation changes under perturbation\.

### A\.2Intrinsic Dimensions

#### A\.2\.12\-Nearest Neighbors

For each data pointii, in this instance an individual token’s activation on layerll\(like in CKA,1≤l≤nl​a​y​e​r1\\leq l\\leq n\_\{layer\}\), we compute distancesr1ir\_\{1\}^\{i\}andr2ir\_\{2\}^\{i\}between the point and its two nearest neighbors within the batch\. Defining

μi=r2ir1i,\\mu\_\{i\}=\\frac\{r\_\{2\}^\{i\}\}\{r\_\{1\}^\{i\}\},we use the maximum likelihood estimator to approximate the intrinsic dimensiondd:

d=N∑ln⁡μi\\displaystyle d=\\frac\{N\}\{\\sum\\ln\\mu\_\{i\}\}whereNNis the number of total points in the batch\.

### A\.3Attention Function Scores

#### A\.3\.1Previous\-Token Score

We report this score as an average of all places in the attention matrix where positioniiattends to positioni−1i\-1\. The attention matrixAAused by the previous token score is computed using clean inputsxx\(α=h⁡\(x\)\\alpha=h\(x\)\), and the overall score is averaged over all inputs:

Previous​\_​Token​\(h\)=𝔼x​\[1L−1​∑i=1L−1αi,i−1\]\.\\displaystyle\\mathrm\{Previous\\\_Token\}\(h\)=\\mathbb\{E\}\_\{x\}\\left\[\\frac\{1\}\{L\-1\}\\sum\_\{i=1\}^\{L\-1\}\\alpha\_\{i,i\-1\}\\right\]\.

#### A\.3\.2Duplicate Score

In order to isolate the duplicate detection signal from previous\-token signal and general semantic associations, we construct an input probe by repeating a randomized sequence ofrrofLLtokens twice\. Then, we evaluate the average attention values inα=h⁡\(r​r\)\\alpha=h\(rr\)where positions from the first sequence attend to their repeated counterpart in the second sequence:

Duplicate⁡\(h\)=1L​∑i=L\+12​Lαi,i−L\.\\displaystyle\\mathrm\{Duplicate\}\(h\)=\\frac\{1\}\{L\}\\sum\_\{i=L\+1\}^\{2L\}\\alpha\_\{i,i\-L\}\.

#### A\.3\.3Induction Score

We use the same repeated random token probe as the duplicate score, but examine the attention between the positions in the repeated sequence and the positions immediately following their counterparts in the earlier sequence\.

Induction⁡\(h\)=1L−1​∑i=L\+12​L−1αi,i−L\+1\.\\displaystyle\\mathrm\{Induction\}\(h\)=\\frac\{1\}\{L\-1\}\\sum\_\{i=L\+1\}^\{2L\-1\}\\alpha\_\{i,i\-L\+1\}\.

### A\.4Attention Entropy Norm Delta

Given a clean inputxxand a perturbed inputx′x^\{\\prime\}, we obtain attention matricesα=h⁡\(x\)\\alpha=h\(x\)andα′=h⁡\(x′\)\\alpha^\{\\prime\}=h\(x^\{\\prime\}\)\. In our experiments, we constructx′x^\{\\prime\}under 30% perturbation for all perturbation types\. Defining entropy at positioniias

H^​\(α\)=1L−1​∑i=1L−1Hi​\(α\)log⁡\(i\+1\)\.\\displaystyle\\widehat\{H\}\(\\alpha\)=\\frac\{1\}\{L\-1\}\\sum\_\{i=1\}^\{L\-1\}\\frac\{H\_\{i\}\(\\alpha\)\}\{\\log\(i\+1\)\}\.
we take the normed entropy over all positions excluding the first \(since the first position does not have a target attention distribution to draw from\)\. The normed entropy divides the total entropy at each location by the maximum possible entropy, to ensure that all entropies are evaluated at comparable scale:

Finally, the*entropy norm delta*over all inputs is given by

Δ​H=𝔼x​\[H^​\(α′\)−H^​\(α\)\]\.\\displaystyle\\Delta\{H\}=\\mathbb\{E\}\_\{x\}\\left\[\\hat\{H\}\(\\alpha^\{\\prime\}\)\-\\hat\{H\}\(\\alpha\)\\right\]\.

## Appendix BAdversarial Shortlist Selection

![Refer to caption](https://arxiv.org/html/2609.03322v1/figures/n_candidates_nll.png)Figure 5:Sequence NLL as a function of HotFlip’s gradient\-shortlist size \(nn\_candidates\), on GPT\-2 at 30% perturbation strength \(n=300n=300\)\.We fix\|S\|=50\|S\|=50based on an ablation over\|S\|∈\{1,5,10,20,50,100\}\|S\|\\in\\\{1,5,10,20,50,100\\\}on GPT\-2 atp=30%p=30\\%\(Figure[5](https://arxiv.org/html/2609.03322#A2.F5)\)\. Mean NLL rises from 8\.91 \(\|S\|=1\|S\|\{=\}1\) to 11\.65 \(\|S\|=100\|S\|\{=\}100\), while the marginal gain per additional candidate declines from 0\.22 between\|S\|=1\|S\|\{=\}1–55to 0\.01 between\|S\|=50\|S\|\{=\}50–100100\. We select\|S\|=50\|S\|=50as a practical balance between attack strength and compute cost rather than as a loss\-maximizing choice\.