DMDIntel: Interpreting Large Language Models via Dynamic Mode Decomposition

# Interpreting Large Language Modelsvia Dynamic Mode Decomposition
Source: [https://arxiv.org/html/2608.13048](https://arxiv.org/html/2608.13048)
## DMDIntel: Interpreting Large Language Models via Dynamic Mode Decomposition

Animesh MukherjeeAffiliation:IIT Kharagpur, IndiaSergey UtyuzhnikovAffiliation:University of Manchester, UK

###### Abstract

In this work, we introduceDMDIntelwhich usesdynamic mode decomposition\(DMD\) to make the predictions made by LLMs in a classification task interpretable\. It develops an input attribution pipeline, that first decomposes the hidden states of an LLM into prominent patterns, also known asmodes, and then associates ranks to the input tokens based on the projection values on those modes\. Rigorous experiments across three datasets and three model families consistently show that the ranked attribution of input tokens obtained usingDMDIntelby far outperforms state\-of\-the\-art techniques such as principal component analysis, integrated gradients andSHAP\.

## 1Introduction

Transformer language modelsvaswani2017attentionredefined the processing of natural language by introducing self\-attention\. Following this, many modifications have been made to the transformer architecture, such as encoder\-only and decoder\-only modelsdevlin2019bert;radford2018improving\. Due to an increase in the complexity of these models, interpretability remains a huge challenge\. A significant body of literature has been dedicated to dissecting and interpreting the internal mechanisms of attention layers in transformer\-based architectures\. Traditional interpretability methods, such as Integrated Gradients \(IG\)sundararajan2017axiomatic,SHAPlundberg2017unified, etc\., are often used for input attribution in LLMs\. Although these methods are effective in identifying “what” the model looks at, they treat the tokens as discrete entities and often overlook the sequential evolution of the latent information, providing a fragmented snapshot rather than a continuous narrative of model’s reasoning\. In addition to attribution, there are some attempts to decode the structural logic of the MLP component of the decoder layer\.geva2021transformersuggests that the MLP module acts as a key\-value memory bank, where the first linear layer is a pattern detector \(keys\) and the second is a value provider that updates the residual stream\.hernandez2023linearitystates that much of the black\-box computation within an MLP can be approximated via linear operations in high\-dimensional space\. However, even these structural insights frequently treat each token’s hidden state as an isolated vector, failing to capture the logic that builds across a sequence\. To address these issues, in this paper, we attempt to take advantage of masked self\-attention, which invokes the sequential nature of decoder\-only LLMs by analyzing hidden states as they evolve token by token\. By viewing these hidden states as trajectories, we can treat the supervised fine\-tuned LLM as a dynamical system where the internal processing resembles a flow in the embedding space\. We use Dynamic Mode Decomposition \(DMD\)schmid2010dynamicto find a surrogate linear operator,𝐀\\mathbf\{A\}, which approximates the complex model underlyingℱ\\mathcal\{F\}\. Specifically, we model the state transition,ht\+1=ℱ⁡\(w≤t\)h\_\{t\+1\}=\\mathcal\{F\}\(w\_\{\\leq t\}\), using a linear relationshipht\+1=𝐀​hth\_\{t\+1\}=\\mathbf\{A\}h\_\{t\}, wherew≤tw\_\{\\leq t\}represents the tokens until the current token indexttandhth\_\{t\}represents the hidden state of the model at the token indextt\. This linear operator is then decomposed usingDMDinto interpretable spatio\-temporal modes\. These modes can be thought of as the low\-dimensional structures that manifest when a model is supervised fine\-tuned for a particular downstream task\. We then use the dominant modes for input attribution to identify the most influential tokens\. Thus, we can think of these tokens as the main drivers of the model’s input processing, as it prepares the output by reading the input tokens\.

Key contributions: Our key contributions in this work are as follows\.

- •We introduceDMDIntel, an input attribution framework for supervised fine\-tuned LLMs that extracts the primary tokens driving a given output classification\. We show that the hidden states of a supervised fine\-tuned model, obtained from thedown\_projof its multi\-layer perceptron \(MLP\) component, show interpretable patterns when decomposed into lower\-dimensional spatio\-temporal modes\. The ranking of tokens by their projections onto theseDMDmodes demonstrates that certain modes serve as robust indicators of the model’s information flow\.
- •We validate the versatility and robustness of our approach by evaluating it across three distinct LLM families of varying parameter scales using three diverse text classification datasets\.
- •To the best of our knowledge, this is the first work to analyze the sequential evolution of hidden states in decoder\-only LLMs as a dynamical system for input attribution\. We demonstrate thatDMDIntelconsistently identifies the highest proportion of ground\-truth tokens while preserving their ranked importance, outperforming baseline methods such asPCA,IG, andSHAP\.

## 2Background

Dynamic Mode Decomposition \(DMD\)schmid2010dynamicwas originally developed in fluid mechanics to extract coherent spatial structures from complex, non\-linear flow fields\.DMDis a purely data\-driven, model\-agnostic technique that requires no prior knowledge of the underlying system governing equations\. By operating directly on time\-series observation snapshots,DMDconstructs a best\-fit linear operator that approximates the non\-linear dynamics of the system\. The resulting eigendecomposition yields a set of spatialDMDmodes—representing the primary spatial structures along with complex eigenvalues that quantify their temporal growth, decay, and oscillation frequencies\. A complete mathematical derivation and algorithmic setup ofDMDis detailed in Appendix[B](https://arxiv.org/html/2608.13048#A2)\.

## 3Related work

Interpretability of LLMs: Various analytical frameworks have been proposed to understand the inner workings of LLMs\. Probing\-based approaches evaluate whether hidden representations capture structural or semantic features by training linear classifiers on top of frozen model layerstenney\-etal\-2019\-bert\. Beyond static probing, mechanistic interpretability seeks to reverse\-engineer specific network circuits, mapping exact weights and attention head paths to functional behaviors such as in\-context pattern matchingolsson2022contextor factual retrievalmeng2022locating\. To observe how these internal features evolve into final predictions, methods such as the LogitLens decode intermediate hidden states directly into the vocabulary space\. This concept is further refined in TunedLensbelrose2023eliciting, which trains linear adapters at each layer to more accurately map representation dynamics to the model’s predictive trajectory\.

DMDapplications: Beyond fluid dynamics,DMDhas been successfully applied to video processing scenarios\.kutz2017dynamicestablishedDMD’s capability for background modeling in video stream data, effectively separating the foreground from the background\.erichson2019compressedutilized a memory\-efficient variant ofDMDto achieve foreground\-background separation in streaming video data\.

DMDfor NLP tasks: In recent works,DMDhas also been used for feature extraction to improve performance in classification tasks\. For example,sachin2019dynamicusesDMDto extract spatio\-temporal features from text representations to capture the evolving sentiment trajectory across a sentence\. Similarly, this dynamical perspective has been successfully applied to multimodal and spoken language tasks; for example,mao2020eigenemoutilizesDMDto derive spectral representations of audio utterances\.vyshnav2020offensiveevaluatesDMDalong with traditional sequence models to isolate consistent and underlying semantic patterns from highly noisy and user\-generated social media text\. Together, these works highlight the efficacy ofDMDin capturing non\-linear dynamics across diverse linguistic modalities\.

## 4TheDMDIntelframework

In this section, we outline theDMDIntelframework which we propose for interpreting the predictions of decoder\-only LLMs when they are used to perform various text classification tasks\. We view the LLMs as a discrete\-time dynamical system, where the evolution of hidden states acts as a first\-order approximation of the LLMℱ\\mathcal\{F\}, such thathk\+1≈ℱ⁡\(hk\)h\_\{k\+1\}\\approx\\mathcal\{F\}\(h\_\{k\}\)\. Subsequently, we useDMD\(see Appendix[B](https://arxiv.org/html/2608.13048#A2)for more details\) to approximateℱ\\mathcal\{F\}, and finally decompose it to get a set of interpretable modes\.

### 4\.1Formulation of the data matrix

As an LLM processes a text sequence, it stores the information in the intermediate hidden states such thatht\+1=ℱ⁡\(w≤t\)h\_\{t\+1\}=\\mathcal\{F\}\(w\_\{\\leq t\}\)\. The information can be considered as the combined representation of the instruction prompt plus the information from the actual input sentence in some non\-linear fashion\. This information is passed through all the layers, before passing through the unembedding matrix, followed by softmax to generate the actual token\. We target a layer, and in that layer, we consider outputs from thedown\_projof the multilayer perceptron\. For an input sequence consisting ofw1,w2,…,wnw\_\{1\},w\_\{2\},\.\.\.,w\_\{n\}tokenized words, we collect the vectors from the selected layer to form the matrixXsuch that

X=\[\|\|\|h1h2⋯hn\|\|\|\]\\textbf\{X\}=\\begin\{bmatrix\}\|&\|&&\|\\\\ h\_\{1\}&h\_\{2\}&\\cdots&h\_\{n\}\\\\ \|&\|&&\|\\end\{bmatrix\}\(1\)where eachhi=ℱ⁡\(w≤i\)h\_\{i\}=\\mathcal\{F\}\(w\_\{\\leq i\}\)\.

### 4\.2Noise removal

We find that the instruction fine\-tuned models encode a significant representation of the prompt template used during fine\-tuning within the hidden representations of each token\. Ashemati2017biasingproves the sensitivity ofDMDto noise in the data, we denoise the data matrix obtained in Eq\.[1](https://arxiv.org/html/2608.13048#S4.E1)by removing the noise inflicted by the prompt template\. For this, we pass an empty prompt consisting of the same exact system and user prompts without the input sentence, and then collect the hidden representation from the same layer from where we pick the token representations\. We call this instructionbias,𝒃\\bm\{b\}\. Thisbiasis then subtracted from eachhih\_\{i\}to obtain the debiased estimate of the hidden representation of the actual tokenwiw\_\{i\}\.

h^i=hi−𝒃\\hat\{h\}\_\{i\}=h\_\{i\}\-\\bm\{b\}\(2\)Thus, our resulting data matrix is the collection of the debiased hidden states of each token\.

𝑿^=\[\|\|\|h^1h^2⋯h^n\|\|\|\]\\boldsymbol\{\\hat\{X\}\}=\\begin\{bmatrix\}\|&\|&&\|\\\\ \\hat\{h\}\_\{1\}&\\hat\{h\}\_\{2\}&\\cdots&\\hat\{h\}\_\{n\}\\\\ \|&\|&&\|\\end\{bmatrix\}\(3\)
![Refer to caption](https://arxiv.org/html/2608.13048v1/llama_fake_cosine.png)\(a\)Llama\-3\.2\-3B\-inst: FE
![Refer to caption](https://arxiv.org/html/2608.13048v1/llama_hate_cosine.png)\(b\)Llama\-3\.2\-3B\-inst: HX
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_fake_cosine.png)\(c\)Qwen3\-4B\-inst: FE
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_sentiment_cosine.png)\(d\)Qwen3\-4B\-inst: SNT
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_hate_cosine.png)\(e\)Qwen3\-4B\-inst: HX
![Refer to caption](https://arxiv.org/html/2608.13048v1/mistral_sentiment_cosine.png)\(f\)Mistral\-7B\-v0\.3\-inst: SNT

Figure 1:Cosine similarity curves𝒞\\mathcal\{C\}across different datasets and models\.xx\-axis shows the layer indices starting from the middle layer of the model till the end\.yy\-axis shows the average cosine similarity of the bias vector with the input token sentences\. FE:FakeEdit, HX:HateXplain, SNT:Sentiment\.
### 4\.3Layer selection

Algorithm 1Heuristic layer selection1:Fine\-tuned LLM

ℱ\\mathcal\{F\}; input sequence

𝐰=\(w1,w2,…,wn\)\\mathbf\{w\}=\(w\_\{1\},w\_\{2\},\\ldots,w\_\{n\}\); instruction bias vector

𝒃\\bm\{b\}\(obtained by passing an empty prompt through

ℱ\\mathcal\{F\}\); cosine threshold

τ=0\.25\\tau=0\.25
2:Selected layer index

ℓ∗\\ell^\{\*\}
3:Let

LmidL\_\{\\mathrm\{mid\}\}denote the middle\-indexed decoder layer of

ℱ\\mathcal\{F\}, and

LlastL\_\{\\mathrm\{last\}\}the final decoder layer\.

4:foreach layer

ℓ∈\{Lmid,Lmid\+1,…,Llast\}\\ell\\in\\\{L\_\{\\mathrm\{mid\}\},\\,L\_\{\\mathrm\{mid\}\}\+1,\\,\\ldots,\\,L\_\{\\mathrm\{last\}\}\\\}do

5:Extract thedown\_projhidden representation

ht\(ℓ\)h\_\{t\}^\{\(\\ell\)\}for each token

wtw\_\{t\}, without debiasing\.

6:Compute the per\-token cosine similarity with the bias vector:

ct\(ℓ\)=⟨ht\(ℓ\),𝒃⟩‖ht\(ℓ\)‖​‖𝒃‖c\_\{t\}^\{\(\\ell\)\}\\;=\\;\\frac\{\\langle h\_\{t\}^\{\(\\ell\)\},\\,\\bm\{b\}\\rangle\}\{\\\|h\_\{t\}^\{\(\\ell\)\}\\\|\\,\\\|\\bm\{b\}\\\|\}
7:Average over all tokens to obtain a layer\-level similarity score:

c¯\(ℓ\)=1n​∑t=1nct\(ℓ\)\\bar\{c\}^\{\(\\ell\)\}\\;=\\;\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}c\_\{t\}^\{\(\\ell\)\}
8:endfor

9:Form the cosine\-similarity curve

𝒞=\(c¯\(ℓ\)\)ℓ=LmidLlast\\mathcal\{C\}=\\bigl\(\\bar\{c\}^\{\(\\ell\)\}\\bigr\)\_\{\\ell=L\_\{\\mathrm\{mid\}\}\}^\{L\_\{\\mathrm\{last\}\}\}\.

10:Restrict the candidate set to layers whose similarity lies below the threshold:

ℒτ=\{ℓ∣c¯\(ℓ\)<τ\}\\mathcal\{L\}\_\{\\tau\}\\;=\\;\\bigl\\\{\\,\\ell\\;\\mid\\;\\bar\{c\}^\{\(\\ell\)\}<\\tau\\,\\bigr\\\}
11:Within

ℒτ\\mathcal\{L\}\_\{\\tau\}, compute the discrete gradient of

𝒞\\mathcal\{C\}:

∇c¯\(ℓ\)=c¯\(ℓ\+1\)−c¯\(ℓ\)\\nabla\\bar\{c\}^\{\(\\ell\)\}\\;=\\;\\bar\{c\}^\{\(\\ell\+1\)\}\-\\bar\{c\}^\{\(\\ell\)\}
12:Identify the layer at the lower end of the steepest ascent in

𝒞\\mathcal\{C\}:

ℓ∗=arg​maxℓ∈ℒτ∇c¯\(ℓ\)\\ell^\{\*\}\\;=\\;\\operatorname\*\{arg\\,max\}\_\{\\ell\\;\\in\\;\\mathcal\{L\}\_\{\\tau\}\}\\;\\nabla\\bar\{c\}^\{\(\\ell\)\}
13:return

ℓ∗\\ell^\{\*\}

Recall that a transformer has multiple layers, and each layer has its own representation of the input tokens\. We define a heuristic to identify the decoder layer whosedown\_projMLP outputs best represent the actual input sentence, balancing input\-token fidelity against instruction\-following bias\. The intuition behind this heuristic is as follows\. Layers close to the middle layer of the transformer,LmidL\_\{\\mathrm\{mid\}\}tend to be nearly orthogonal to𝒃\\bm\{b\}, indicating that their representations are driven primarily by the actual input tokens\. Asℓ\\ellincreases towardLlastL\_\{\\mathrm\{last\}\}\(last transformer layer\), the cosine similarity also rises, reflecting a shift in the model’s focus toward instruction\-following rather than input content\. The selected layerℓ∗\\ell^\{\*\}sits at the inflection point of this curve𝒞\\mathcal\{C\}– the point of highest local gradient still below the cosine similarity thresholdτ=0\.25\\tau=0\.25– where the model strikes the best balance between encoding input\-token semantics and attending to the instruction prompt\. We verify that this heuristic is consistent across all model families and datasets considered in this work \(see example curves𝒞\\mathcal\{C\}in Figure[1](https://arxiv.org/html/2608.13048#S4.F1)for different datasets and models discussed in Section[5](https://arxiv.org/html/2608.13048#S5)\)\. The procedure based on this heuristic is formalized in Algorithm[1](https://arxiv.org/html/2608.13048#alg1), and the layers selected for different model and dataset settings are detailed in Table[1](https://arxiv.org/html/2608.13048#S4.T1)

Table 1:Selected optimal layer indices across different datasets and model families\.
### 4\.4Input token attribution

We use PyDMD,Demo2018, a robust Python package, which consists of implementations of variousDMDalgorithms, to obtain the approximate surrogateℱ\\mathcal\{F\}of the LLM using the debiased hidden state matrix\. The modes obtained after the eigendecomposition of the linear Koopman operatorAcan be interpreted as capturing low\-dimensional structures underlying the evolution of hidden representations\. We interpret these structures to reflect linguistic and semantic regularities, such as syntactic patterns, grammatical dependencies, and features relevant to the predictive behavior of the model\. Apart from the basicDMDalgorithm we also use the higher order dynamic mode decomposition \(HoDMD\)doi:10\.1137/15M1054924\. WhileDMDfocuses on finding a linear operator satisfying the Markovian assumption \(ht\+1=𝐀​hth\_\{t\+1\}=\\mathbf\{A\}h\_\{t\}\),HoDMDimplements time embedding \(see Appendix[C](https://arxiv.org/html/2608.13048#A3)for more details\)\. In this way, a Hankel matrix is constructed from the original snapshot data to model the system such that each hidden state is treated as a linear combination of theddprevious time\-steps, whereddis the delay parameter\. To implementHoDMD, we stackddsuccessive snapshots into a higher\-dimensional representation, which allows the model to capture long\-term dynamics and temporal dependencies that standardDMDmight miss\. The expanded snapshot matrixℋ\\mathcal\{H\}looks as follows

ℋ1N−d=\[h1ℓ∗h2ℓ∗…hN−dℓ∗h2ℓ∗h3ℓ∗…hN−d\+1ℓ∗⋱hdℓ∗hd\+1ℓ∗…hNℓ∗\]\\mathcal\{H\}\_\{1\}^\{N\-d\}=\\begin\{bmatrix\}h\_\{1\}^\{\\ell^\{\*\}\}&h\_\{2\}^\{\\ell^\{\*\}\}&\\dots&h^\{\\ell^\{\*\}\}\_\{N\-d\}\\\\ h\_\{2\}^\{\\ell^\{\*\}\}&h\_\{3\}^\{\\ell^\{\*\}\}&\\dots&h\_\{N\-d\+1\}^\{\\ell^\{\*\}\}\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ h\_\{d\}^\{\\ell^\{\*\}\}&h\_\{d\+1\}^\{\\ell^\{\*\}\}&\\dots&h\_\{N\}^\{\\ell^\{\*\}\}\\end\{bmatrix\}\(4\)Our pipeline then proceeds by applying standardDMDto this augmented matrix\. First, we perform an SVD onℋ\\mathcal\{H\}to reduce dimensionality while preserving dominant features\. Next, we compute the Koopman eigenvalues and eigenvectors from the reduced\-rank operator to identify the system’s modes\. Mode ranking: Similar toPCA, where components are ranked by the amount of variance explained,DMD/HoDMDmodes can be ranked according to different measures of dynamical significance\. We employ two ranking criteria: \(i\) the magnitude of the initial modal amplitudesROWLEY2009, and \(ii\) the average magnitude of the modal amplitudes over the temporal trajectoryCRMECA\_2014\_\_342\_6\-7\_410\_0to get the most relevant modes \(top\-kk\) for our task\. In all our experiments, we considerk=5k=5modes, both when experimenting withDMD\(and other similar baselines likePCA\)\. Token ranking: To find how each token contributes to those modes, we consider the state change \(Δ​htℓ∗\\Delta h\_\{t\}^\{\\ell^\{\*\}\}\) which the tokenwtw\_\{t\}adds after it has been processed, as a proxy for the hidden state representation of that token, and then calculate its scalar projection magnitude on each of the modes\. We then sum the projection magnitude values for each mode for all the topkkmodes to obtain a projection score for that token\.

Δ​htℓ∗\\displaystyle\\Delta h\_\{t\}^\{\\ell^\{\*\}\}=htℓ∗−ht−1ℓ∗\\displaystyle=h\_\{t\}^\{\\ell^\{\*\}\}\-h\_\{t\-1\}^\{\\ell^\{\*\}\}\(5\)αt,i\\displaystyle\\alpha\_\{t,i\}=\|⟨Δ​htℓ∗,ϕi⟩\|‖ϕi‖,i=1,…,k\\displaystyle=\\frac\{\\left\|\\langle\\Delta h\_\{t\}^\{\\ell^\{\*\}\},\\phi\_\{i\}\\rangle\\right\|\}\{\\\|\\phi\_\{i\}\\\|\},\\quad i=1,\\dots,k\(6\)st\\displaystyle s\_\{t\}=∑i=1kαt,i\\displaystyle=\\sum\_\{i=1\}^\{k\}\\alpha\_\{t,i\}\(7\)where,ϕi\\phi\_\{i\}is a selected mode,⟨⋅⟩\\langle\\cdot\\ranglerepresents the dot product of two vectors,kkrepresents the number of top modes considered, andsts\_\{t\}is the projected score obtained for the particular token indextt\. Finally, we rank the tokens according to their decreasing order of projection scores to obtain the ranked input attributions for each token\.

## 5Experimental setup

Models: In this work, we fine\-tune three different LLMs,Llama\-3\.2\-3B\-instgrattafiori2024llama,Qwen3\-4B\-instyang2025qwen3, andMistral\-7B\-v0\.3\-instjiang2023mistral\. We perform supervised instruction fine\-tuning of the model in full precision\. The prompt templates used for the fine\-tuning are provided in the Appendix\. Datasets: For this work, we use three text classification datasets \(see Appendix[D](https://arxiv.org/html/2608.13048#A4)for more details\) as follows – \(i\)Sentiment: sentiment classificationzhang2015character, \(ii\)FakeEdit: fake news detectionnakamura2020fakeddit, and \(iii\)HateXplain: hateful content classificationmathew2021hatexplain\. We use the training split of each of these datasets to perform the supervised fine\-tuning of the three chosen models\. Finally, we use 1000 different sentences from the test set of each dataset, equally divided in different input size buckets, to evaluate the efficacy of our method on small as well as large input sentences\.

Baselines: We compare our method withIGsundararajan2017axiomatic,SHAPlundberg2017unified\(more specifically, theGradientSHAPimplementation\) andPCA\. Note thatPCAis computed directly on the transpose of debiased hidden state matrix,𝑿^⊤\\hat\{\\bm\{X\}\}^\{\\top\}, such that each row is treated as an observation \(token hidden states\) and each column represents the hidden dimension, to obtain a set of principal components that maximizes the variance in the collection of representation vectors\. The token attribution inPCAthen follows the same procedure asDMD, mentioned in Eq\.[5](https://arxiv.org/html/2608.13048#S4.E5)to[7](https://arxiv.org/html/2608.13048#S4.E7)\.

Table 2:Examples of input sentences and their corresponding token\-level ground truth \(GT\) rankings generated by GPT\-4\.1\.Metrics: We report three metrics: \(i\) average matched count, \(ii\) Rank\-Biased Overlap \(RBO\)10\.1145/1852102\.1852106and \(iii\) recall@kk\. The average matched count gives the expected number of tokens matched from the GT, it does not take into account the number of tokens GT has for a particular sentence\. RBO compares the rank of the retrieved token with the token’s rank in the GT and penalizes tokens that are assigned a higher rank than in the GT\. RBO is defined as follows,

RBOEXT​\(S,T,p,k\)=\(1−p\)​∑d=1kAd⋅pd−1\+Xkk⋅pk\\textbf\{RBO\}\_\{\\text\{EXT\}\}\(S,T,p,k\)=\(1\-p\)\\sum\_\{d=1\}^\{k\}A\_\{d\}\\cdot p^\{d\-1\}\+\\frac\{X\_\{k\}\}\{k\}\\cdot p^\{k\}\(8\)whereSSandTTare two retrieved lists,ppis the persistence factor,AdA\_\{d\}is the agreement between two lists at depthdd,kkis the top\-kkcut\-off value, andXkX\_\{k\}is the overlap count betweenSSandTTat depthkk\. For our evaluations, we choosep=0\.95p=0\.95\. Recall@kkmeasures the percentage of GT tokens retrieved\. It is sensitive to both the number of retrieved tokens and the number of tokens in the GT\. Ground truth: For each dataset, ground truth tokens responsible for a prediction were obtained using OpenAI GPT\-4\.1achiam2023gpt, where the model was prompted to rank individual tokens from input sentences based on their importance to the output label\. A maximum of 20 tokens were ranked for each input, apart from stop words, and the model was instructed to stop the ranking early if there were fewer than 20 influential tokens\. Table[2](https://arxiv.org/html/2608.13048#S5.T2)lists an example sentence from each dataset’s class label, and the corresponding ground truth \(GT\) returned by GPT\-4\.1\. The GTs are ranked in order of the most influential to the least influential in determining the class label of the input sentence\. In order to test the goodness of the labelling, two experienced annotators manually rank 100 input sentences from each dataset which is then compared with the GPT\-4\.1 annotated rankings\. We first observe that the two experienced annotators agree very well in terms of the RBO scores for all the three datasets\. The scores are as follows – negative sentiment: 0\.64, positive sentiment: 0\.72, hateful text: 0\.62, and fake text: 0\.61\. Next, we find that the human annotations and GPT\-4\.1 annotations also align well\. In particular, RBO values obtained for the three datasets are negative sentiment: 0\.60, positive sentiment: 0\.70, hateful text: 0\.54, and fake text: 0\.60\. The slightly lower agreement forHateXplaincorresponds to the known subjectivity of hate speech annotation in the literature\.

Table 3:Experimental results for theSentimentdataset across three different model families\. MC: Matched count,HoDMD\-avgamp:HoDMDwith averaged amplitude ranking\. Best results are inboldand the second best areunderlined\.Table 4:Experimental results for theHateXplainandFakeEditdatasets across three different model families\. The attributions are computed only for the class of interest \(i\.e\., hateful forHateXplainand fake forFakeEdit\)\. MC: Matched count,DMD\-amp:DMDwith amplitude ranking\. Best results are inboldand the second best areunderlined\.
## 6Results

We evaluate our framework using 1,000 samples per dataset, partitioned into distinct buckets based on input length to assess performance across varying context sizes\. For theSentimentdataset, samples are categorized into three ranges: 15–40, 40–70, and 70–100 tokens\. For theHateXplainandFakeEditdatasets, we utilize two buckets: 15–40 and 40–70 tokens\. As noted earlier, we evaluate four combinations as follows – \(i\) standardDMDwith amplitude\-based ranking, \(ii\) standardDMDwith time\-averaged amplitude ranking, \(iii\)HoDMDwith amplitude\-based ranking using an adaptive delay parameter, and \(iv\)HoDMDwith time\-averaged amplitude ranking using an adaptive delay parameter\. In each setting \(model \+ dataset\), we report the metrics obtained for the best performing combination above alongside the baselines \(see Appendix[D](https://arxiv.org/html/2608.13048#A4)for full results\)\.

### 6\.1Dataset based results

Sentiment classification: For sentiment classification, we retrieve the top 20 tokens for both sentiments and then compare them with the GT\. We find thatHoDMD\(see Table[3](https://arxiv.org/html/2608.13048#S5.T3)\), paired with mode ranking through the averaged amplitude, consistently outperforms all other configurations across the three evaluated models\. This setup generally exceeds traditional baselines, includingPCA,IG, andSHAPin most metrics\. Since sentiment is typically expressed through multi\-word phrases rather than isolated tokens, the use of delay factors in theHoDMDsetup provides the necessary context to capture these dynamics more effectively than the standardDMD\. HateXplain: For this dataset, the standardDMDusing amplitude\-based ranking proves to be the most effective configuration for all models\. As each data point inHateXplaincontains very few sentences with more than 10 GT tokens, we calculate the metrics for the top 10 retrieved tokens\. Note that here we compute the attributions on for the class of interest \(i\.e\., hateful class\)\. ForLlama\-3\.2\-3B\-instandQwen3\-4B\-inst, theDMD\-based attribution generally outperforms the baseline methods \(Table[4](https://arxiv.org/html/2608.13048#S5.T4)\)\. ForMistral\-7B\-v0\.3\-inst, our method is the second best\. SinceHateXplainrelies on the specific token\-level annotations for hateful content, we believe that the state changes triggered by these individual tokens are well\-captured by the standardDMD\. This suggests that the evolution of hidden representations in this context follows a near\-Markovian process that does not require the extended memory ofHoDMD\. FakeEdit: The results forFakeEditalign closely with those ofHateXplain\(Table[4](https://arxiv.org/html/2608.13048#S5.T4)\)\. Because the dataset consists of Reddit posts where specific keywords—such as the names of political figures—often dictate the “fake” status of a post, the importance is concentrated on individual tokens\. Here, again, the attributions are computed for the fake class\. Consequently, the standardDMDwith amplitude ranking achieves the best performance, outperforming all baseline methods in recall@10\. The importance ranking of the tokens may not align best with the GT, but our method retrieves the most amount of tokens from GT as measured by recall\. This confirms that for tasks where localized token information is critical, standard linear approximations are highly effective\.

### 6\.2Qualitative results

ModelMethodTop\-ranked Tokens \(Attribution\)“dont buy this phone this phone makes a constant high static shrieking noise i bought this phone to use on a regularly scheduled long conference call and was totally unable to use it its going directly in the trash"\[Negative\]Llama\-3\.2\-3B\-instGT1\. trash, 2\. unable, 3\. shrieking, 4\. static, 5\. noise, 6\. constant, 7\. dont, 8\. buy, 9\. totally, 10\. directlyDMDIntel1\. going, 2\. trash, 3\. dont, 4\. unable, 5\. conference, 6\. shrieking, 7\. makes, 8\. bought, 9\. regularly, 10\. high, 11\. scheduled, 12\. use, 13\. long, 14\. directly, 15\. phone, 16\. constant, 17\. call, 18\. static, 19\. totallyPCA1\. going, 2\. unable, 3\. conference, 4\. shrieking, 5\. makes, 6\. high, 7\. bought, 8\. trash, 9\. phone, 10\. regularly, 11\. constant, 12\. static, 13\. long, 14\. use, 15\. noise, 16\. directlyIG1\. scheduled, 2\. dont, 3\. conference, 4\. use, 5\. totally, 6\. trash, 7\. directly, 8\. going, 9\. use, 10\. static, 11\. call, 12\. unable, 13\. noise, 14\. buy, 15\. regularly, 16\. longSHAP1\. trash, 2\. directly, 3\. noise, 4\. high, 5\. scheduled, 6\. use, 7\. constant, 8\. going, 9\. static, 10\. dont, 11\. phone, 12\. totallyQwen3\-4B\-instGT1\. trash, 2\. unable, 3\. shrieking, 4\. static, 5\. noise, 6\. constant, 7\. dont, 8\. buy, 9\. totally, 10\. directlyDMDIntel1\. going, 2\. directly, 3\. trash, 4\. dont, 5\. regularly, 6\. makes, 7\. buy, 8\. static, 9\. unable, 10\. bought, 11\. scheduled, 12\. use, 13\. totally, 14\. long, 15\. phone, 16\. shrieking, 17\. conference, 18\. constant, 19\. noisePCA1\. dont, 2\. buy, 3\. going, 4\. bought, 5\. makes, 6\. regularly, 7\. shrieking, 8\. trash, 9\. constant, 10\. scheduled, 11\. static, 12\. noise, 13\. directly, 14\. use, 15\. conference, 16\. totally, 17\. call, 18\. long, 19\. unableIG1\. use, 2\. dont, 3\. unable, 4\. regularly, 5\. totally, 6\. conference, 7\. scheduled, 8\. use, 9\. noise, 10\. directly, 11\. going, 12\. call, 13\. constant, 14\. phone, 15\. bought, 16\. high, 17\. static, 18\. phone, 19\. trashSHAP1\. use, 2\. high, 3\. conference, 4\. totally, 5\. dont, 6\. regularly, 7\. makes, 8\. phone, 9\. buy, 10\. scheduled, 11\. directly, 12\. shrieking“you are retarded and do not know what reply mean or are a shitty troller suck a bag of ni\*\*er d\*cks fa\*\*ot"\[Hateful\]Llama\-3\.2\-3B\-instGT1\. ni\*\*er, 2\. fa\*\*ot, 3\. retarded, 4\. shitty, 5\. d\*cksDMDIntel1\. fa\*\*ot, 2\. d\*cks, 3\. suck, 4\. ni\*\*er, 5\. bag, 6\. troller, 7\. shitty, 8\. retarded, 9\. meanPCA1\. suck, 2\. reply, 3\. know, 4\. retarded, 5\. troller, 6\. bag, 7\. d\*cks, 8\. shitty, 9\. mean, 10\. fa\*\*ot, 11\. ni\*\*erIG1\. shitty, 2\. fa\*\*ot, 3\. troller, 4\. mean, 5\. know, 6\. bag, 7\. ni\*\*erSHAP1\. fa\*\*ot, 2\. retarded, 3\. know, 4\. ni\*\*ger, 5\. bag, 6\. shittyQwen3\-4B\-instGT1\. ni\*\*er, 2\. fa\*\*ot, 3\. retarded, 4\. shitty, 5\. d\*cksDMDIntel1\. fa\*\*ot, 2\. d\*cks, 3\. bag, 4\. suck, 5\. ni\*\*er, 6\. troller, 7\. reply, 8\. mean, 9\. shitty, 10\. know, 11\. retardedPCA1\. retarded, 2\. d\*cks, 3\. fa\*\*ot, 4\. bag, 5\. suck, 6\. know, 7\. reply, 8\. mean, 9\. troller, 10\. shitty, 11\. ni\*\*erIG1\. bag, 2\. ni\*\*er, 3\. retarded, 4\. fa\*\*ot, 5\. suck, 6\. troller, 7\. know, 8\. d\*cks, 9\. shittySHAP1\. fa\*\*ot, 2\. retarded, 3\. reply, 4\. ni\*\*er, 5\. d\*cksTable 5:Attribution forLlama\-3\.2\-3B\-instandQwen3\-4B\-instmodels on negative sentiment and hate speech sentences\.Some of the representative qualitative results are noted in Table[5](https://arxiv.org/html/2608.13048#S6.T5)\.DMDIntelconsistently demonstrates the strongest alignment with GT attributions across both models and sentence types, ranking the most semantically relevant tokens near the top of its lists\. For the negative sentence,DMDIntelplaces “trash” and “unable” within its topfourforLlama\-3\.2\-3B\-inst, while for the hate speech it recovers all five GT tokens within its topninefor both models\. Notably,DMDIntelalso exhibits strong cross\-model consistency – its top five tokens for hate speech are nearly identical betweenLlama\-3\.2\-3B\-instandQwen3\-4B\-inst, differing only in minor reordering\.SHAPperforms competitively, recovering several high\-priority GT tokens in compact lists: for the hate speech sentence it ranks “fa\*\*ot” and “retarded” in its top two for both models, and for the negative sentence it places “trash” first forLlama\-3\.2\-3B\-inst\. However,SHAPtends to surface contextually plausible but GT\-absent tokens \(e\.g\. “high”, “conference”, “scheduled”\) at the expense of key sentiment markers such as “unable” and “shrieking”\.PCAshows moderate but uneven alignment, capturing several GT tokens yet struggling to prioritize the most discriminative ones \(e\.g\., ranking “ni\*\*er”, the highest GT token, last in the hate speech sentence forLlama\-3\.2\-3B\-inst\)\.IGshows the weakest performance overall, with non\-salient tokens frequently appearing at the top of its rankings and instability manifesting as duplicate token entries \(e\.g\., “use” appears twice in itsQwen3\-4B\-instnegative sentence list\)\. Beyond ranking quality, the methods also differ in coverage and consistency\.DMDIntelandPCAproduce longer attribution lists, surfacing a broader set of contributing tokens, whileSHAPandIGyield shorter and more concentrated lists\. This compactness is a strength forSHAP, which tends to maintain reasonable precision within its smaller set, but a liability forIG, which risks omitting relevant tokens such as “noise” and “constant” in the negative sentence\. Across models,PCAandIGexhibit greater sensitivity to the underlying model’s representations — for instance,PCA’s top\-ranked tokens for the negative sentence differ substantially betweenLlama\-3\.2\-3B\-instandQwen3\-4B\-inst— whereasDMDIntelandSHAPremain comparatively stable\. Taken together, these observations suggest thatDMDInteloffers the best balance of GT recall, coverage, and cross\-model robustness, withSHAPbeing a reasonable but less comprehensive alternative, andPCAandIGlargely lagging behind\.

### 6\.3Sensitivity to layer selection

Here we investigate how sensitiveDMDIntelis to the layer selected \(see Algorithm[1](https://arxiv.org/html/2608.13048#alg1)\) for obtaining our results\. We report the recall values for each of the datasets and models in Table[6\.3](https://arxiv.org/html/2608.13048#S6.SS3)\. We note thatDMDIntelis quite stable, and the results usingℓ∗−1\\ell^\{\*\}\-1orℓ∗\+1\\ell^\{\*\}\+1are very similar to those usingℓ∗\\ell^\{\*\}\. In fact, as long as the cosine similarity of the bias and the selected layer is close to that between the bias andℓ∗\\ell^\{\*\}, the results remain largely unchanged\.

Table 6:Comparison of accuracy drop \(%\) and confidence score drop undert​o​p−ktop\\mathchar 45ktoken masking across the models onSentimentandHateXplaindatasets\. Higher drops \(↑\\uparrow\) indicate more faithful attributions\.Table 7:Layer\-wise sensitivity analysis\.Δmax\\Delta\_\{\\text\{max\}\}represents the maximum DMD performance difference observed from layerℓ∗\\ell^\{\*\}to eitherℓ∗−1\\ell^\{\*\}\-1orℓ∗\+1\\ell^\{\*\}\+1\.### 6\.4Fidelity based analysis

To evaluate the faithfulness of our input attribution framework, we conduct a token\-masking experiment following established perturbation benchmarks\. Specifically, we measure the drop in model accuracy and output confidence when masking the top\-kkmost influential tokens as identified by each attribution method, with whitespace characters\. As reported in Table[6](https://arxiv.org/html/2608.13048#S6.T6), our approach consistently ranks first or second in both accuracy and confidence drops across all evaluated model families and datasets, confirming its robust capability to locate the most critical input features\. Notably, while gradient\-based baselines likeIGachieve high accuracy drops onHateXplainby isolating explicit target terms, they often focus strictly on a narrow subset of toxic keywords\. This behavior is corroborated by higher RBO scores alongside lower recall values in Table[4](https://arxiv.org/html/2608.13048#S5.T4), indicating thatIGoverlooks implicit, contextually essential tokens\. In contrast, our method captures a broader, more cohesive set of influential features while maintaining competitive perturbation performance\.

## 7Conclusion

In this paper, we proposed a framework for interpreting the predictions of fine\-tuned LLMs by analyzing the evolution of their hidden states during sequence processing\. By treating the model’s internal mechanics as a dynamical system, we demonstrate thatDMDprovides a more effective lens for input attribution than traditional methods\. Our findings suggest that this dynamical perspective offers a better and more robust way to understand how LLMs represent and process complex information compared to state\-of\-the\-art methods\.

## Acknowledgment

The authors gratefully acknowledge the financial and institutional support provided by the Joint PhD Programme between the Indian Institute of Technology Kharagpur \(IIT Kharagpur\) and The University of Manchester\.

## 8Limitations

In this section, we discuss potential limitations of our approach in this work for interpreting LLMs\. First, we only fine\-tune and evaluate LLMs for classification tasks, where there are very few tokens \(1\-2\) generated as a response to the input query\. While this makes our approach efficient for identifying key tokens from the input when only one to two tokens are generated as output, it is still a challenge to find out the influential tokens from the input if more tokens are generated as these tokens also get fed to the model in an auto\-regressive manner to generate the next token\. For example, in machine translation from English to French, one can find the tokens that dominate the low dimensional structures for the first few French tokens generated, but as these French tokens are also fed to the model to generate the complete translation, formulating theDMDmatrix in this case becomes somewhat difficult\. So accurately mapping the English tokens responsible for French translated tokens becomes a challenging task and is potential future work\.

Second, in this work we only explore supervised instruction fine\-tuned LLMs for simple downstream tasks, but the general purpose instruction models released by the companies remains unexplored in this study\. These instruction tuned models are base models fine\-tuned on huge datasets for instruction following tasks, and identifying key patterns in their input processing is also a potential future work\.

## References

## Appendix AUse of AI assistance

We employed proprietary LLMs solely for obtaining the ground truth labels for the datasets, editorial purposes, including refining grammar, spelling, word choice, and overall clarity of the manuscript\.

## Appendix BDynamic mode decomposition

Dynamic mode decomposition was introduced inschmid2010dynamic, to extract dynamic patterns from a flow field in fluid mechanics\.DMDworks on observational data, without any underlying assumptions of the dynamical system\. Consider a set of observations obtained from a dynamical system, represented by the matrix𝑿1N\\bm\{X\}\_\{1\}^\{N\},X\_1^N = \[x\_1, x\_2, …, x\_N\] where each column vector𝒙i\\bm\{x\}\_\{i\}represents anit​hi^\{th\}observation of sizem\. Next, the assumption is that, there exists a linear mappingAwhich connects the observationxix\_\{i\}toxi\+1x\_\{i\+1\}as follows x\_i\+1 =Ax\_i In the matrix notation we can write the above equation as,

𝑿2N≈A​𝑿1N−1\\bm\{X\}\_\{2\}^\{N\}\\approx\\textbf\{A\}\\bm\{X\}\_\{1\}^\{N\-1\}\(9\)where,𝑿1N−1∈m×n−1\\bm\{X\}\_\{1\}^\{N\-1\}\\in m\\times n\-1, is the matrix containing the firstN−1N\-1snapshots, and𝑿2N\\bm\{X\}\_\{2\}^\{N\}is the matrix shifted by one time\-step\. Generally, the spatial dimension of the systems is much larger than the temporal dimension,m≫nm\\gg nand hence Eq[9](https://arxiv.org/html/2608.13048#A2.E9)can be solved using the singular\-value decomposition \(SVD\):

X=U​Σ​V∗X=U\\Sigma V^\{\*\}\(10\)where \* represents complex conjugate transpose,U∈ℂn×rU\\in\\mathbb\{C\}^\{n\\times r\},Σ∈ℂr×r\\Sigma\\in\\mathbb\{C\}^\{r\\times r\}, andV∈ℂr×nV\\in\\mathbb\{C\}^\{r\\times n\}\. Hence

A=𝑿2N​V​Σ−1​U∗\\textbf\{A\}=\\bm\{X\}\_\{2\}^\{N\}V\\Sigma^\{\-1\}U^\{\*\}\(11\)as computing this can be computationally expensive, we projectAonto the lower dimensional subspace defined by the eigenvectors of the snapshot matrix,A~=U∗​A​U\\tilde\{\\textbf\{A\}\}=U^\{\*\}\\textbf\{A\}Uand then again define theDMDproblem as,

x~t\+1=A~​x~t\\tilde\{\\textbf\{x\}\}\_\{t\+1\}=\\tilde\{\\textbf\{A\}\}\\tilde\{\\textbf\{x\}\}\_\{t\}\(12\)After computingA~\\tilde\{\\textbf\{A\}\}, the eigenvector\-decomposition is done and then the lower dimensional eigenvectors are projected back to the original spatial dimension, to approximate the eigenvectors ofA\.

A~​𝐖=𝐖​Λ\\tilde\{\\textbf\{A\}\}\\mathbf\{W\}=\\mathbf\{W\}\\Lambda\(13\)𝚽=𝑿2N​V​Σ−1​𝐖\\boldsymbol\{\\Phi\}=\\bm\{X\}\_\{2\}^\{N\}V\\Sigma^\{\-1\}\\mathbf\{W\}\(14\)The eigenvectors obtained from Eq[13](https://arxiv.org/html/2608.13048#A2.E13), are known asDMDmodes, and they represent the dominant patterns which construct the flow of the dynamical system\. Each eigenvectorϕi\\phi\_\{i\}, is associated with a complex eigenvalue which represents the strength and the frequency of oscillations of that eigenvector\.

## Appendix CHigher order dynamic mode decomposition

HoDMDexploits a delay\-embedding approach which helps it capture temporal dependencies across multiple snapshots\. Consider a set of observations𝑿1N=\[x1,x2,…,xN\]\\bm\{X\}\_\{1\}^\{N\}=\[x\_\{1\},x\_\{2\},\\dots,x\_\{N\}\]\. The core assumption ofHoDMDis that there exists a higher\-order linear relationship such that the observationxi\+dx\_\{i\+d\}is a linear combination of the previousddsnapshots:

xi\+d≈𝐀0​xi\+𝐀1​xi\+1\+⋯\+𝐀d−1​xi\+d−1x\_\{i\+d\}\\approx\\mathbf\{A\}\_\{0\}x\_\{i\}\+\\mathbf\{A\}\_\{1\}x\_\{i\+1\}\+\\dots\+\\mathbf\{A\}\_\{d\-1\}x\_\{i\+d\-1\}\(15\)whereddis the design parameter representing the number of delays\. This is solved by constructing a Hankel snapshot matrixℋ1N−d\+1\\mathcal\{H\}\_\{1\}^\{N\-d\+1\}by stackingddsuccessive snapshots into a higher\-dimensional representation as follows\.

ℋ1N−d\+1=\[x1x2…xN−d\+1x2x3…xN−d\+2⋱xdxd\+1…xN\]\\mathcal\{H\}\_\{1\}^\{N\-d\+1\}=\\begin\{bmatrix\}x\_\{1\}&x\_\{2\}&\\dots&x\_\{N\-d\+1\}\\\\ x\_\{2\}&x\_\{3\}&\\dots&x\_\{N\-d\+2\}\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ x\_\{d\}&x\_\{d\+1\}&\\dots&x\_\{N\}\\end\{bmatrix\}\(16\)In this augmented space, the system can be treated as a first\-order dynamical system:

ℋ2N−d\+1≈𝒜​ℋ1N−d\\mathcal\{H\}\_\{2\}^\{N\-d\+1\}\\approx\\mathbf\{\\mathcal\{A\}\}\\mathcal\{H\}\_\{1\}^\{N\-d\}\(17\)whereℋ2N−d\+1\\mathcal\{H\}\_\{2\}^\{N\-d\+1\}is the Hankel matrix shifted by one time\-step\. To efficiently compute the operator𝒜\\mathbf\{\\mathcal\{A\}\}, the SVD is performed on the Hankel matrix similar toDMD\.

ℋ1N−d=𝒰​Σ​𝒱∗\\mathcal\{H\}\_\{1\}^\{N\-d\}=\\mathcal\{U\}\\Sigma\\mathcal\{V\}^\{\*\}\(18\)The high\-dimensional operator is then projected onto the lower\-dimensional subspace defined by the SVD modes of the Hankel matrix, yielding the reduced\-order operator𝐀~\\tilde\{\\mathbf\{A\}\}:

𝐀~=𝒰∗​ℋ2N−d\+1​𝒱​Σ−1\\tilde\{\\mathbf\{A\}\}=\\mathcal\{U\}^\{\*\}\\mathcal\{H\}\_\{2\}^\{N\-d\+1\}\\mathcal\{V\}\\Sigma^\{\-1\}\(19\)The reduced operator is then eigendecomposed as:

𝐀~​𝐖=𝐖​Λ\\tilde\{\\mathbf\{A\}\}\\mathbf\{W\}=\\mathbf\{W\}\\Lambda\(20\)TheHoDMDmodes are obtained by projecting these eigenvectors back\. Since the eigenvectors𝐖\\mathbf\{W\}exist in the augmentedm×dm\\times ddimensional space, the spatialDMDmodes𝚽\\boldsymbol\{\\Phi\}are recovered by extracting the firstmmcomponents \(the first block\) of the projected eigenvectors:

𝚽=𝒰1:m,:𝐖\\boldsymbol\{\\Phi\}=\\mathcal\{U\}\_\{1:m,:\}\\mathbf\{W\}\(21\)Each modeϕi\\phi\_\{i\}in𝚽\\boldsymbol\{\\Phi\}represents a spatio\-temporal pattern, while the corresponding eigenvalue inΛ\\Lambdadescribes the temporal evolution \(frequency and growth/decay rate\) of that specific pattern\.

## Appendix DExperimental setting and detailed results

We fine\-tune all the models on a single NVIDIA H100 GPU, and we utilize NVIDIA L40 GPUs for model inference and execution of the input attribution pipelines\.

### D\.1Sentiment analysis

We use Amazon polarity dataset which has reviews extracted from Amazon and their corresponding labels aspositiveandnegativezhang2015character\. All the three LLMs used in this study, were fully fine\-tuned on 150,000 train samples in full precision\. The test accuracy achieved on 10,000 test samples was around 97% for all the three models with very minor variations\.

The instruction prompt used for fine\-tuning the LLMs and also used to get thebiasvector without passing the input sentence for sentiment analysis is as follows\.

![[Uncaptioned image]](https://arxiv.org/html/2608.13048v1/sentiment_instruct_template.png)We use the chat template format provided by HuggingFace to define the above chat template, and then HuggingFace handles the tokenization of this template internally\. To obtain the vector corresponding to the entire representation of the instruction template, we find the review index \_start and then use the index before that to extract the bias vector from the hidden states\. Table[8](https://arxiv.org/html/2608.13048#A4.T8)shows the comparison study of ourDMDattribution pipeline of all possible ranking variants against the baselines\. From the metrics, it is evident that the modes whose amplitudes persist for longer, capture the sentiment better than those whose initial amplitude has a very high value\.

### D\.2HateXplain

FromHateXplainmathew2021hatexplain, we only consider attribution towards hateful sentences, although the model was trained for binary classification withhatefulandnormallabels\. After fine\-tuning on 10,000 samples for 2 epochs,Llama\-3\.2\-3B\-instachieved an accuracy of 82%,Qwen3\-4B\-instachieved an accuracy of 83\.2% andMistral\-7B\-v0\.3\-instachieved an accuracy of 84% on the test set consisting of 2,000 samples\.

The instruction prompt used for fine\-tuning the LLMs is as follows:

![[Uncaptioned image]](https://arxiv.org/html/2608.13048v1/hateful_instruct_template.png)The same prompt was also used for the generation of bias vector from the layer considered in the experiments\. Table[8](https://arxiv.org/html/2608.13048#A4.T8)shows full comparison study between differentDMDalgorithms and ranking techniques\. As the dataset contains the hateful triggers as single words or a few collection of words, it can be well approximated by vanillaDMD, and the modes whose amplitude starts high prove to be useful for finding the correct attribution tokens\.

### D\.3FakeEdit

We useFakeEditdataset, which contains posts taken from Reddit and categorized into two labels;fakeandnot\-fake\. These reviews are mostly post titles with short to medium input length sentences\. We fine\-tune the LLMs on a train set of 15,000 samples for 2 epochs and achieve an accuracy of 84% forLlama\-3\.2\-3B\-inst, 85\.6% forQwen3\-4B\-instand 86% forMistral\-7B\-v0\.3\-inston a test set of 2,000 samples\. The prompt template used in fine\-tuning and generating the bias vector is as follows\.

![[Uncaptioned image]](https://arxiv.org/html/2608.13048v1/fake_instruct_template.png)The observations forFakeEditare mostly similar toHateXplain, asFakeEditalso contains individual trigger words which can determine the “fakeness” of a sentence\. Consequently,DMDwith amplitude ranking performs the best among all variants and baselines\.

### D\.4IGandSHAP

We use Captum librarykokhlikyan2020captumto implement the pipelines forIGandSHAP\. In both implementations, we calculate the attributions on the embedding layer for each token, which directly measures the impact of the input tokens on the generated output\.

Table 8:Detailed Attribution results across models and datasets\. Best results are noted inbold\.Llama\-3\.2\-3B\-inst\- Sentiment

Qwen3\-4B\-inst\- Sentiment

Mistral\-7B\-v0\.3\-inst\- Sentiment

Llama\-3\.2\-3B\-inst\- HateXplain and FakeEdit

Qwen3\-4B\-inst\- HateXplain and FakeEdit

Mistral\-7B\-v0\.3\-inst\- HateXplain and FakeEdit

## Appendix EAnalysis of eigenvalues and the modes

The temporal evolution of the dynamical system is governed by the discrete\-time eigenvaluesλi\\lambda\_\{i\}obtained from the eigendecomposition of the linear operator𝐀\\mathbf\{A\}\. These eigenvalues provide critical information regarding the stability, growth, and oscillatory nature of their correspondingDMDmodesϕi\\phi\_\{i\}\. The position of the eigenvalues relative to the unit circle in the complex plane determines the asymptotic behavior of the modes categorized as follows\.

- •Steady modes \(\|λi\|=1\|\\lambda\_\{i\}\|=1\): Eigenvalues falling exactly on the unit circle correspond to pure oscillations with constant amplitude\. These represent the steady\-state dynamics of the system\.
- •Stable modes \(\|λi\|<1\|\\lambda\_\{i\}\|<1\): Eigenvalues located within the unit circle represent physically damped or transient dynamics\. The closer the eigenvalue is to the origin, the more rapid the decay of the mode as the sequence progresses\.
- •Unstable modes \(\|λi\|\>1\|\\lambda\_\{i\}\|\>1\): Eigenvalues outside the unit circle indicate exponential growth, often associated with diverging hidden state representations or numerical instabilities in the local flow\.

Based on this theory, we conduct an analysis of the eigenvalues and the dynamics of the hidden states of the representative sentences from the test set of all datasets for each model\. We find that for all the sentences, the hidden states produced by the model inducedecayingandstablemodes\. While some modes persist forever without any oscillations \(mode 0 in the plots\), there exist some modes that decay to zero amplitude before the sentence even finishes, and some modes still oscillate without ever converging to zero amplitude\.

Figure[2](https://arxiv.org/html/2608.13048#A5.F2)provides a comprehensive comparison of eigenvalue distributions and mode dynamics across various model and dataset configurations\. Each sub\-figure consists of two complementary plots: the left plot displays the unit circle \(with real and imaginary components on the axes\), while the right plot illustrates the corresponding temporal dynamics\. In the dynamics plot, thexx\-axis represents the token index \(time\-step\) and theyy\-axis tracks the real part of the mode’s amplitude\. These dynamics plots effectively show how the influence of specific modes fluctuates throughout the processing of a sentence\. To ensure a fair comparison, we use the same sample input sentences for every model\-dataset pair\. The results reveal that even when processing identical inputs, the selected modes and their behaviors vary significantly across models\. This divergence is likely due to differences in hidden state dimensions, which force each model to encode and process information in its own distinct way\. This observation is further supported by the results in Table[8](https://arxiv.org/html/2608.13048#A4.T8)of this Appendix\. AlthoughPCAperformance fluctuates significantly between different models even on the same dataset, our optimizedDMDconfiguration remains remarkably stable\. This consistency suggests thatDMDis less sensitive to architectural variations, providing more reliable attribution metrics across diverse model scales and hidden state dimensions\.

![Refer to caption](https://arxiv.org/html/2608.13048v1/llama_neg_dmd_analysis.png)\(g\)Llama\-3\.2\-3B\-inst: Negative sentiment
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_neg_dmd_analysis.png)\(h\)Qwen3\-4B\-inst: Negative sentiment
![Refer to caption](https://arxiv.org/html/2608.13048v1/mistral_neg_dmd_analysis.png)\(i\)Mistral\-7B\-v0\.3\-inst: Negative sentiment
![Refer to caption](https://arxiv.org/html/2608.13048v1/llama_pos_dmd_analysis.png)\(j\)Llama\-3\.2\-3B\-inst: Positive sentiment
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_pos_dmd_analysis.png)\(k\)Qwen3\-4B\-inst: Positive sentiment
![Refer to caption](https://arxiv.org/html/2608.13048v1/mistral_pos_dmd_analysis.png)\(l\)Mistral\-7B\-v0\.3\-inst: Positive sentiment
![Refer to caption](https://arxiv.org/html/2608.13048v1/llama_hate_dmd_analysis.png)\(m\)Llama\-3\.2\-3B\-inst:HateXplain
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_hate_dmd_analysis.png)\(n\)Qwen3\-4B\-inst:HateXplain
![Refer to caption](https://arxiv.org/html/2608.13048v1/mistral_hate_dmd_analysis.png)\(o\)Mistral\-7B\-v0\.3\-inst:HateXplain
![Refer to caption](https://arxiv.org/html/2608.13048v1/llama_fake_dmd_analysis.png)\(p\)Llama\-3\.2\-3B\-inst:FakeEdit
![Refer to caption](https://arxiv.org/html/2608.13048v1/qwen_fake_dmd_analysis.png)\(q\)Qwen3\-4B\-inst:FakeEdit
![Refer to caption](https://arxiv.org/html/2608.13048v1/mistral_fake_dmd_analysis.png)\(r\)Mistral\-7B\-v0\.3\-inst:FakeEdit

Figure 2:Eigenvalue distributions and mode dynamics across various model and dataset configurations\.
## Appendix FError analysis

We also conduct an error analysis of how good is the approximation𝐀\\mathbf\{A\}found byDMDby reconstructing the hidden states for sample sentences and then measuring the Frobenius norm between the original hidden states and the reconstructed states\. The reconstruction of the snapshot matrix𝑿^orig\\hat\{\\bm\{X\}\}\_\{\\text\{orig\}\}is achieved by utilizing the extractedDMDmodes𝚽\\boldsymbol\{\\Phi\}, the diagonal matrix of eigenvalues𝚲\\bm\{\\Lambda\}, and the vector of initial amplitudes𝜶\\boldsymbol\{\\alpha\}\. The approximated state at timettis given by:

h^t=∑i=1rϕi​λit−1​αi=𝚽​𝚲t−1​𝜶\\hat\{h\}\_\{t\}=\\sum\_\{i=1\}^\{r\}\\phi\_\{i\}\\lambda\_\{i\}^\{t\-1\}\\alpha\_\{i\}=\\boldsymbol\{\\Phi\}\\bm\{\\Lambda\}^\{t\-1\}\\boldsymbol\{\\alpha\}\(22\)
For the entire sequence ofNNsnapshots, the reconstructed matrix𝑿^recons\\hat\{\\bm\{X\}\}\_\{\\text\{recons\}\}is represented as:

𝑿^recons=𝚽​diag​\(𝜶\)​\[1λ1…λ1N−11λ2…λ2N−1⋱1λr…λrN−1\]\\hat\{\\bm\{X\}\}\_\{\\text\{recons\}\}=\\boldsymbol\{\\Phi\}\\text\{diag\}\(\\boldsymbol\{\\alpha\}\)\\begin\{bmatrix\}1&\\lambda\_\{1\}&\\dots&\\lambda\_\{1\}^\{N\-1\}\\\\ 1&\\lambda\_\{2\}&\\dots&\\lambda\_\{2\}^\{N\-1\}\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ 1&\\lambda\_\{r\}&\\dots&\\lambda\_\{r\}^\{N\-1\}\\end\{bmatrix\}\(23\)
To evaluate the fidelity of this approximation, we calculate the relative reconstruction error using the Frobenius norm\|\|⋅\|\|F\|\|\\cdot\|\|\_\{F\}\. The errorϵ\\epsilonis defined as the ratio of the norm of the residual to the norm of the original snapshot matrix:

ϵ=‖𝑿^orig−𝑿^recons‖F‖𝑿^orig‖F\\epsilon=\\frac\{\|\|\\hat\{\\bm\{X\}\}\_\{\\text\{orig\}\}\-\\hat\{\\bm\{X\}\}\_\{\\text\{recons\}\}\|\|\_\{F\}\}\{\|\|\\hat\{\\bm\{X\}\}\_\{\\text\{orig\}\}\|\|\_\{F\}\}\(24\)
Table[9](https://arxiv.org/html/2608.13048#A6.T9)represents the error values for the models and the datasets\. We find that the reconstruction error rate is very low in the order of10−210^\{\-2\}, suggesting that the linear approximation of the LLM usingDMDis quite accurate\.

Table 9:Reconstruction errorϵ\\epsilon\.Table 10:Detailed efficiency evaluation comparing peak GPU memory usage \(GB\) and per\-sample attribution computation time \(seconds\) across model families and benchmark datasets\. ForSentimentdataset, execution times are reported for negative/positive context passes\.
## Appendix GComputational efficiency and resource utilization

To assess the computational efficiency of our proposed framework, we benchmark its average execution time per input and peak GPU memory footprint against established baselines\. All timing and memory profiling experiments were conducted on NVIDIA L40 GPUs using single\-precision floating\-point format \(float32\) for model parameters and internal vector representations\. As detailed in Table[10](https://arxiv.org/html/2608.13048#A6.T10), our approach achieves superior efficiency, requiring the lowest processing time and minimal GPU memory overhead\. This performance gain primarily stems from our architectural design: while standard gradient\- and perturbation\-based techniques likeIGandSHAPrequire dozens to hundreds of forward and backward passes per input sequence, our method extracts attributions non\-intrusively via a single forward pass\. Consequently, this yields a substantial reduction in inference latency and resource consumption\.

## Appendix HAblation studies

### H\.1Results without denoising

To assess the importance of removing the noise inflicted by the empty chat prompt template, we studied the metrics without the removal\. We find that the metrics drop significantly for sentiment analysis, there is a slight drop forHateXplainbut the drop in metrics is not so significant forFakeEdit\. Detailed results for the three models and the datasets forDMDIntelwithout the noise removal are presented in the Tables[11](https://arxiv.org/html/2608.13048#A8.T11)and[12](https://arxiv.org/html/2608.13048#A8.T12)\. This reinforces that the noise removal step not only removes the noise, but also ensures that the hidden state snapshots contain the information which the specific token adds to the residual stream\.

Table 11:Evaluation results of our proposed method \(DMDIntelHoDMD−a​v​g​a​m​p\{\}^\{\\textsc\{HoDMD\}\-avgamp\}\) onSentimentdataset across Negative and Positive sentiment classes for three model families without denoising the hidden states\.Table 12:Evaluation results of our proposed method \(DMDIntelDMD−a​m​p\{\}^\{\\textsc\{DMD\}\-amp\}\) on theHateXplainandFakeEditdatasets across three model families without denoising the hidden states\.
### H\.2Thresholdτ=0\.5\\tau=0\.5

Raising the similarity threshold to0\.50\.5restricts layer selection strictly to representations that exhibit a higher alignment with the model’s instruction\-following dynamics\. Intuitively, enforcing a stricter cutoff filters out lower\-level contextual signals, which is expected to yield a corresponding decline in attribution performance metrics\. Our empirical observations largely align with this expectation, albeit with domain\-specific nuances\. For theFakeEditdataset, we observe a minor reduction across evaluation metrics\. OnHateXplain, performance remains largely stable, accompanied by a subtle increase in RBO scores\. In contrast, for theSentimentdataset across both positive and negative sentiment classes, raising the threshold induces a pronounced drop in metric values forQwen3\-4B\-instandMistral\-7B\-v0\.3\-inst, whereasLlama\-3\.2\-3B\-instmaintains relatively resilient performance\. Complete results across all configurations are reported in Tables[13](https://arxiv.org/html/2608.13048#A8.T13)and[16](https://arxiv.org/html/2608.13048#A8.T16)\.

Table 13:Comparative evaluation results for Sentiment Analysis across negative and positive classes with thresholdτ=0\.5\\tau=0\.5\.
### H\.3Varying thekkvalue in top\-kktokens for calculation of metrics

We check the consistency of the results when we select top\-1010ranked tokens with the GT for theSentimentdataset, and top\-2020ranked tokens with GT forFakeEditandHateXplaindatasets\. We find that the results remain consistent in theSentimentdataset, while in that ofFakeEditandHateXplainthe metrics start to converge for top\-2020ranked tokens as there are very few input sentences from these datasets which have GT tokens more than 10\. Detailed results are presented in Tables[14](https://arxiv.org/html/2608.13048#A8.T14)and[15](https://arxiv.org/html/2608.13048#A8.T15)\.

Table 14:Results for top\-1010ranked tokens in case ofSentimentdataset\.Table 15:Results forHateXplainandFakeEditfor top\-2020ranked tokens\.
### H\.4Selecting top\-77modes

We examine the effect of mode subspace dimensionality by projecting token representations onto the top\-77modes compared to top\-55in our original setup\. As reported in Table[17](https://arxiv.org/html/2608.13048#A8.T17), selecting additional modes leads to a noticeable drop in performance metrics forDMDIntel\. This performance decrease likely stems from noisy higher\-order modes that introduce non\-label\-intensive tokens into the top attributions\. In contrast,PCAshows slight, though statistically marginal, performance gains when the number of modes is increased\.

Table 16:Comparative evaluation results forHateXplainandFakeEditdatasets, with thresholdτ=0\.5\\tau=0\.5\.MethodNegative sentimentPositive sentimentMC@20↑\\uparrowRBO@20↑\\uparrowRecall@20↑\\uparrowMC@20↑\\uparrowRBO@20↑\\uparrowRecall@20↑\\uparrowLlama\-3\.2\-3B\-instPCA5\.280\.240\.665\.990\.270\.73DMDIntelHoDMD−a​v​g​a​m​p\{\}^\{\\textsc\{HoDMD\}\-avgamp\}5\.210\.230\.666\.040\.260\.74Qwen3\-4B\-instPCA5\.460\.250\.686\.080\.280\.75DMDIntelHoDMD−a​v​g​a​m​p\{\}^\{\\textsc\{HoDMD\}\-avgamp\}5\.340\.240\.676\.100\.270\.75Mistral\-7B\-v0\.3\-instPCA5\.270\.240\.665\.760\.250\.71DMDIntelHoDMD−a​v​g​a​m​p\{\}^\{\\textsc\{HoDMD\}\-avgamp\}5\.180\.230\.655\.880\.250\.72Table 17:Selecting top\-7 modes for calculating the attribution for sentiment analysis dataset\.

## Appendix ILabel\-intensive token alignment vs\. decision faithfulness

To systematically evaluate the semantic relevance of identified tokens, we employ GPT\-4\.1 to annotate input sequences with label\-oriented ground\-truth rationale tokens\. For any LLM fine\-tuned on downstream classification, these label\-intensive tokens intuitively represent the core semantic anchors driving the model’s target class selection\. Our empirical findings demonstrate thatDMDsuccessfully isolates the low\-dimensional latent subspaces onto which these label\-intensive tokens yield high projection values, revealing their prominent role within the model’s implicit representation dynamics\. Crucially, we distinguish betweensemantic coverageandexclusive causal dependency\. Identifying these dominant tokens does not imply that the model relies solely on them to synthesize its output, nor does it suggest that non\-labeled contextual tokens are irrelevant to the decision trajectory\. As evidenced by our fidelity\-by\-masking experiments \(Section[6\.4](https://arxiv.org/html/2608.13048#S6.SS4)\)—particularly on theHateXplaindataset where gradient\-based baselines likeIGexhibit sharper accuracy and confidence drops—DMDprioritizes broad, contextually rich label\-intensive patterns over narrow, isolated toxic keywords\. Consequently, whileIGaggressively degrades task performance by removing single high\-leverage tokens, our approach captures a more comprehensive, contextually aligned feature set that reflects the broader latent mechanisms governing sequence processing\.