Probabilistic Attribution For Large Language Models

# Probabilistic Attribution For Large Language Models
Source: [https://arxiv.org/html/2605.21726](https://arxiv.org/html/2605.21726)
\\nameShilpika\\emailshilpika@anl\.gov \\addrArgonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA\\nameCarlo Graziani\\emailcgraziani@anl\.gov \\addrMathematics and Computer Science Division Argonne National Laboratory Lemont, IL, USA\\nameBethany Lusch\\emailblusch@anl\.gov \\addrArgonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA\\nameVenkatram Vishwanath\\emailvenkat@anl\.gov \\addrArgonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA\\nameMichael E\. Papka\\emailpapka@anl\.gov \\addrArgonne Leadership Computing Facility Argonne National Laboratory Lemont, IL, USA \\addrDepartment of Computer Science University of Illinois Chicago Chicago, IL, USA

###### Abstract

The generative nature of Large Language Models \(LLMs\) is reflected in the conditional probabilities they compute to sample each response token given the previous tokens\. These probabilities encode the distributional structure that the model learns in training and exploits in inference\. In this work, we use these probabilities to situate LLMs within the mathematical theory of stochastic processes\. We use this framework to design a model\-agnostic probabilistic token attribution measure, using Bayes rule to invert the next\-token log\-probabilities so as to capture the model’s internal representation of the distribution over token sequences\. The representation is independent of the model’s computational structure\. This representation yields the conditional probability of the response given the prompt, and of the response given the prompt with a token marginalized away\. Our attribution score is the log of the ratio of these probabilities\. We further compute the entropies of a single prompt’s token distributions, conditioned on the remaining context\. The interplay between entropy and attribution score sheds light on LLM behavior\. We evaluate 8 models across 7 prompts and investigate anomalies, token sensitivity, response stability, model stability, and training convergence, thereby improving interpretability and guiding users to focus on uncertain or unstable parts of the generation\.

Keywords:AI, explainability, explainable AI, XAI, interpretability, entropy, probability theory

## 1Introduction

The growing impact of AI on scientific modeling mandates a newly\-rigorous approach to analyzing what AI models do, how they do it, and what uncertainties should be associated with AI\-based predictions and decisions\. Key concerns include appropriate model sizing \(which directly impacts the energy cost of training\), output validity/correctness, and output uncertainty, among others\. These present substantial challenges, because the complexity of the models precludes the kind of direct interpretability that attends first\-principles mathematical/statistical models, making it necessary to address such questions indirectly\.

To enable the understanding of the decision\-making process of AI models, particularly large language models, several techniques, frameworks, and methodologies have been developed, and these are referred to as Explainable Artificial Intelligence \(XAI\)\. XAI provides transparency into how inputs to “black box” AI model generate a response\. The motivation for XAI arises from the need to foster trust, regulation, accountability, fairness, and safety in AI applications, especially in high\-stakes domains like healthcare, finance, and law, where opaque decision\-making can have serious ethical and social consequences\(Phillipset al\.,[2021](https://arxiv.org/html/2605.21726#bib.bib2); Rudin,[2019](https://arxiv.org/html/2605.21726#bib.bib3)\)\.

XAI encompasses methods that can be broadly categorized into model\-specific approaches, which are tailored to certain classes of models \(for example, saliency maps and gradient\-based visualizations in deep neural networks\), and model\-agnostic approaches, which can be applied universally regardless of the underlying algorithm\(Barredo Arrietaet al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib4)\)\. Model\-agnostic methods such as SHAP \(SHapley Additive exPlanations\) and LIME \(Local Interpretable Model\-agnostic Explanations\) rely on input–output behavior rather than internal architecture, making them particularly versatile across different machine learning models\(Lundberg and Lee,[2017](https://arxiv.org/html/2605.21726#bib.bib24); Ribeiroet al\.,[2016b](https://arxiv.org/html/2605.21726#bib.bib23)\)\.

In this work, we situate the inference operations of large language models \(LLMs\) within the mathematical theory of stochastic processes\. Using this framework, we develop a probabilistic attribution score \(AS\) for large\-language models that is model\-agnostic\. TheASdetermines the importance of tokens in a prompt given a generated response\. LLMs are fundamentally autoregressive, meaning they generate text one token at a time by sampling a computed probability distribution for the next output token conditioned on all the previous ones\. LLMs work because they successfully infer statistical properties of token sequences from training corpuses and exploit those properties to make near\-optimal prompt\-response decisions\. This process is directly grounded in the chain rule of probability, which allows the joint probability of a sequence of random variables to be factorized into a product of conditional probabilities\. Stochastic process theory provides a useful mathematical framework for understanding those operations in a model\-agnostic fashion that permits easy comparison between different types of LLMs\. This framework also permits useful probabilistic analyses of LLM responses, of whichASis an example\.

Our work has the following contributions:

- C1\.ASis model\-agnostic and is applied to text\-generation tasks, although it can be extended to any NLP\-based task\.
- C2\.Our method is parameter\-free, which makes it less sensitive to arbitrary user choices and improves reproducibility and objectivity, making it suitable for hyperparameter\-tuning of other XAI methods\.
- C3\.Our contextual and replacement entropy are measures of model stability, and can indicate how well these models converged to their training data distribution\.
- C4\.AShas built\-in sensitivity, as it is based on statistical first principles and thus evaluates the correctness of the explainable results as part of its core methodology\.

The remainder of this paper is organized as follows\. The Background section reviews the key concepts, prior work, and theoretical foundations relevant to the study\. The LLMs as Stochastic Processes, Experiments, and Evaluation sections then describe the proposed approach, experimental setup, implementation details, comparative analysis, and the criteria used to assess performance\. Finally, the Conclusion summarizes the main findings, discusses their implications, and outlines potential directions for future research\. The code is available at:https://github\.com/sshilpika/probabilistic\-attribution\-score/\.

## 2Background: Explainable AI for NLP and Large Language Models

Explainable artificial intelligence \(XAI\) in natural language processing \(NLP\) has evolved from post hoc interpretability methods for linear models and early neural architectures into a diverse methodological landscape tailored to Transformer encoders and contemporary large language models\(Luo and Specia,[2024](https://arxiv.org/html/2605.21726#bib.bib32); Rogerset al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib30)\)\. Current research spans three broad levels: local explanations of individual predictions, global analyses of model behavior and internalized knowledge, and mechanistic interpretability aimed at identifying the internal computational structures that give rise to model outputs\(Luo and Specia,[2024](https://arxiv.org/html/2605.21726#bib.bib32); Elhageet al\.,[2021](https://arxiv.org/html/2605.21726#bib.bib31)\)\.

Early work in NLP XAI focused on local feature\-attribution methods designed to estimate which input tokens or n\-grams most influenced a model’s prediction\. LIME represented a major advance by explaining predictions through locally fitted interpretable surrogate models, often using perturbation\-based token deletion\(Ribeiroet al\.,[2016b](https://arxiv.org/html/2605.21726#bib.bib23)\)\. Closely related Shapley\-value\-based methods, particularly SHAP, provided a more principled additive framework for attribution and became widely adopted in text\-based interpretability pipelines\(Lundberg and Lee,[2017](https://arxiv.org/html/2605.21726#bib.bib24)\)\. For neural NLP models, gradient\-based methods such as Integrated Gradients became especially influential due to their computational scalability and axiomatic grounding\(Sundararajanet al\.,[2017](https://arxiv.org/html/2605.21726#bib.bib25)\)\. Toolkits such as Captum further standardized the implementation and comparison of attribution methods in PyTorch\-based workflows\(Kokhlikyanet al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib26)\)\. However, it soon became clear that token\-level attributions could be unstable under paraphrase, and sensitive to perturbation strategy\(Sinhaet al\.,[2021](https://arxiv.org/html/2605.21726#bib.bib15)\)\.

These limitations motivated the development of rationale\-based explanations, which identify spans of text intended to justify a prediction\. This line of work was consolidated by benchmarks such as ERASER, which distinguished plausibility, or agreement with human\-annotated rationales, from faithfulness, or the extent to which a rationale genuinely supports the model’s prediction\(DeYounget al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib27)\)\. This distinction has become foundational in LLM explainability, where persuasive explanations may still fail to reflect the actual computational basis of a response\(Luo and Specia,[2024](https://arxiv.org/html/2605.21726#bib.bib32)\)\.

Attention\-based interpretability introduced another major debate\. Because attention maps are visually intuitive, they were initially treated as natural explanations of Transformer behavior\. However, Jain and Wallace showed that attention weights often correlate weakly with other importance measures and that substantially different attention distributions can yield similar predictions\(Jain and Wallace,[2019](https://arxiv.org/html/2605.21726#bib.bib28)\)\. Subsequent work argued that the explanatory value of attention depends on the definition of explanation and the evaluation protocol employed\(Wiegreffe and Pinter,[2019](https://arxiv.org/html/2605.21726#bib.bib29)\)\.

In parallel, global interpretability research examined what linguistic knowledge models encode, where it is stored, and how it evolves across layers\. This tradition, often referred to as “BERTology,” employed probing classifiers, representational analyses, and behavioral tests to characterize syntactic and semantic information in model representations\(Rogerset al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib30)\)\. More recently, mechanistic interpretability has extended this agenda by attempting to reverse\-engineer Transformers into circuits, features, and information pathways\(Elhageet al\.,[2021](https://arxiv.org/html/2605.21726#bib.bib31)\)\.

In the LLM era, explainability must also address prompt sensitivity, self\-generated rationales, and retrieval provenance in retrieval\-augmented systems\(Luo and Specia,[2024](https://arxiv.org/html/2605.21726#bib.bib32)\)\. Overall, the field can be organized along several dimensions: intrinsic versus post hoc methods, local versus global scope, the object of explanation, and the criteria of evaluation\. Across these paradigms, the central challenge remains unchanged: producing explanations that are not merely plausible, but causally faithful, stable, and useful in real\-world deployment\(Luo and Specia,[2024](https://arxiv.org/html/2605.21726#bib.bib32); DeYounget al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib27)\)\. Our work is model\-agnostic and does not attempt to ascribe anything relating to “knowledge representation” or “model decision process” because it only concerns itself with text\-distributional properties of LLM inference\.

## 3LLMs as Stochastic Processes

Our approach to LLMs is probabilistic, and explicitly designed to draw a veil over the computational transformer model details\. Instead, we focus here on what the transformer architecture explicitly purports to model: conditional probabilities of tokens, conditioned on the tokens that preceded them in the token sequence\. In effect, a trained LLM defines a probability distribution over texts, which are sequences of tokens\.

Viewed in this light, one can recognize a connection between LLMs and a venerable subject in mathematical statistics: the theory of stochastic processes\. This connection is interesting, because it offers the possibility of situating LLMs within a well\-developed body of mathematical theory, and of exploiting that theory to gain insight into LLMs and their outputs\.

We remind readers of the definition of such objects\. A*stochastic process*is defined as a collection of random variables\[Xt:t∈T\]\[X\_\{t\}:t\\in T\]on a probability space\(Ω,ℱ,Pr\)\(\\Omega,\\mathcal\{F\},\\mathrm\{Pr\}\), whereTTis an arbitrary index set\(Billingsley,[1995](https://arxiv.org/html/2605.21726#bib.bib8), p\. 482\)\. Recall further the elements of a probability space:Ω\\Omegais a set of outcomes, called the*sample space*,ℱ\\mathcal\{F\}is aσ\\sigma\-field of sets of elements ofΩ\\Omega\(the measurable sets\), andPr:ℱ↦\[0,1\]\\mathrm\{Pr\}:\\mathcal\{F\}\\mapsto\[0,1\]is a probability measure satisfyingPr​\(Ω\)=1\\mathrm\{Pr\}\(\\Omega\)=1,Pr​\(∅\)=0\\mathrm\{Pr\}\(\\emptyset\)=0, andPr​\(∪kAk\)=∑kP​\(Ak\)\\mathrm\{Pr\}\\left\(\\cup\_\{k\}A\_\{k\}\\right\)=\\sum\_\{k\}P\(A\_\{k\}\)for any countable collection of disjoint setsAk∈ℱA\_\{k\}\\in\\mathcal\{F\}\.

In the context of LLMs, the index set is the natural numbers including zero,T=ℕ0T=\\mathbb\{N\}\_\{0\}\. The random variables correspond to the tokens issued during LLM inference at each location of a text sequence\. To further press the correspondence between the operations of an LLM and stochastic process theory, it is necessary to introduce some notation\.

### 3\.1Notation

The objects of interest here are sequences of language tokens drawn from the distribution over texts learned by an LLM during training\. We denote position in a token sequence by a Greek index such asμ\\mu, distinguishing such indices from Roman indices such asllthat denote the identity of a token\. The size of the vocabulary is fixed and denoted byVV, so that a token ID \(TID\) indexllsatisfies0≤l<V0\\leq l<V\.

We will denote random variables by upper\-case letters, and their realizations by lower\-case letters\. So, in particular, the symbolLμL\_\{\\mu\}represents a random variable describing the distribution over possible TID values of the token occurring at positionμ\\muin a sequence, while the symbollμl\_\{\\mu\}\(0≤lμ<V0\\leq l\_\{\\mu\}<V\) represents a realization ofLμL\_\{\\mu\}\.

A finite\-length token sequence will be represented by a boldface letter\. So, for example, we may write random variables such as𝑳=\[L0,L1,…,LM−1\]\\bm\{L\}=\\left\[L\_\{0\},L\_\{1\},\\ldots,L\_\{M\-1\}\\right\]to represent sequences of random variables of lengthMM\. Similarly, we write𝒍=\[l0,l1,…,lM−1\]\\bm\{l\}=\\left\[l\_\{0\},l\_\{1\},\\ldots,l\_\{M\-1\}\\right\]for a realized sequence of TIDs of lengthMM\. Note that the elements of such finite\-length sequences need not be contiguous in sequential indexμ\\mu, so that, for example𝑳=\[L5,L9,L22\]\\bm\{L\}=\\left\[L\_\{5\},L\_\{9\},L\_\{22\}\\right\]is a valid finite\-length sequence of random variables, and may be realized by corresponding assignments𝒍=\[l5,l9,l22\]\\bm\{l\}=\\left\[l\_\{5\},l\_\{9\},l\_\{22\}\\right\]\. We denote the length of a finite\-length sequence𝑳\\bm\{L\}by\|𝑳\|\|\\bm\{L\}\|\.

We may now further amplify the correspondence between the operations of LLMs and the above definition of a stochastic process\. The infinite\-length sequence of random variablesℒ≡\[Lμ:μ∈𝒩0\]=\[L0,L1,…\]\\mathcal\{L\}\\equiv\[L\_\{\\mu\}:\\mu\\in\\mathcal\{N\}\_\{0\}\]=\[L\_\{0\},L\_\{1\},\\ldots\]corresponds to the collection\[Xt:t∈T\]\[X\_\{t\}:t\\in T\]in the definition of a stochastic process\. The sample spaceΩ\\Omegain the definition corresponds to the set of all possible infinite sequencesω=\(l0,l1,l2,…\)\\omega=\(l\_\{0\},l\_\{1\},l\_\{2\},\\ldots\)of TIDs\.

As usual in probability theory, an “event” is a set belonging to theσ\\sigma\-field of measurable setsℱ\\mathcal\{F\}, which by the definition of aσ\\sigma\-field is closed under set complementation and countable set unions\. We constructℱ\\mathcal\{F\}as follows: A set expression such as\{𝑳=𝒍\}\\left\\\{\\bm\{L\}=\\bm\{l\}\\right\\\}describes the event “The realization of the finite\-length sequence𝑳\\bm\{L\}is𝒍\\bm\{l\},” and corresponds to the set of all sequencesℒ\\mathcal\{L\}such that the corresponding TID assignments hold, while all other elements ofℒ\\mathcal\{L\}are unspecified \(i\.e\. unobserved\)\. Denote byFFthe collection of all such sets, their finite unions,Ω\\Omega, and the empty set∅\\emptyset\.FFis closed under complementation and finite unions, and is therefore a field\. We defineℱ\\mathcal\{F\}as theσ\\sigma\-field generated byFF,ℱ=σ​\(F\)\\mathcal\{F\}=\\sigma\(F\)\. It is, by definition, the smallest of allσ\\sigma\-fields containingFF\(Billingsley,[1995](https://arxiv.org/html/2605.21726#bib.bib8), pp\. 21\-22\)\.

With this standard set notation for events, the event describing “eventAA*and*eventBB” is succinctly writtenA∩BA\\cap B, whereas “eventAA*or*eventBB” isA∪BA\\cup B\.

In what follows, it will be convenient to have a notation for certain types of subsequences\. With the length\-MMsequence𝑳=\[L0,L1,…,LM−1\]\\bm\{L\}=\\left\[L\_\{0\},L\_\{1\},\\ldots,L\_\{M\-1\}\\right\], we will write𝑳<μ≡\[L0,…,Lμ−1\]\\bm\{L\}\_\{<\\mu\}\\equiv\\left\[L\_\{0\},\\ldots,L\_\{\\mu\-1\}\\right\],𝑳≤μ≡\[L0,…,Lμ\]\\bm\{L\}\_\{\\leq\\mu\}\\equiv\\left\[L\_\{0\},\\ldots,L\_\{\\mu\}\\right\],𝑳\>μ≡\[Lμ\+1,…,LM−1\]\\bm\{L\}\_\{\>\\mu\}\\equiv\\left\[L\_\{\\mu\+1\},\\ldots,L\_\{M\-1\}\\right\], and similarly for the sequence realizations𝒍<μ\\bm\{l\}\_\{<\\mu\},𝒍≤μ\\bm\{l\}\_\{\\leq\\mu\},𝒍\>μ\\bm\{l\}\_\{\>\\mu\}\. The notation𝑳−Lμ≡𝑳<μ∩𝑳\>μ\\bm\{L\}\-L\_\{\\mu\}\\equiv\\text\{$\\bm\{L\}\_\{<\\mu\}$\}\\cap\\bm\{L\}\_\{\>\\mu\}then represents the finite subsequence over theLνL\_\{\\nu\},0≤ν<M0\\leq\\nu<M,ν≠μ\\nu\\neq\\mu\. This helps us represent events such as\{𝑳−Lμ=𝒍−lμ\}\\left\\\{\\bm\{L\}\-L\_\{\\mu\}=\\bm\{l\}\-l\_\{\\mu\}\\right\\\}, the set of sequences wherein all tokens in positionsν<M\\nu<Mexceptlμl\_\{\\mu\}have realized values, while the values oflμl\_\{\\mu\}and oflνl\_\{\\nu\},ν≥M\\nu\\geq Mare undetermined\.

### 3\.2The Probability Measure

The striking thing about the view of LLMs as stochastic processes is that the probability measurePr\\mathrm\{Pr\}that enters the probability space\(Ω,ℱ,Pr\)\(\\Omega,\\mathcal\{F\},\\mathrm\{Pr\}\)is provided by the LLM itself, constructed from its training data\.

LLMs operate sequentially, yielding vectors of categorical probabilities for the next token in a sequence conditioned on all previous tokens\. That is, they supply

Pr​\(\{Lμ=lμ\}\|\{𝑳<μ=𝒍<μ\}\)\\mathrm\{Pr\}\\left\(\\left\\\{L\_\{\\mu\}=l\_\{\\mu\}\\right\\\}\|\\left\\\{\\bm\{L\}\_\{<\\mu\}=\\bm\{l\}\_\{<\\mu\}\\right\\\}\\right\)\(1\)where theLμL\_\{\\mu\}are the tokens that make up a sequence\. By the chain rule of probability one can directly form response sequence probabilities via

Pr​\(\{𝑳≥μ=𝒍≥μ\}\|\{𝑳<μ=𝒍<μ\}\)\\displaystyle\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{L\}\_\{\\geq\\mu\}=\\bm\{l\}\_\{\\geq\\mu\}\\right\\\}\|\\left\\\{\\bm\{L\}\_\{<\\mu\}=\\bm\{l\}\_\{<\\mu\}\\right\\\}\\right\)=\\displaystyle=Pr​\(\{Lμ=lμ\}\|\{𝑳<μ=𝒍<μ\}\)\\displaystyle\\mathrm\{Pr\}\\left\(\\left\\\{L\_\{\\mu\}=l\_\{\\mu\}\\right\\\}\|\\left\\\{\\bm\{L\}\_\{<\\mu\}=\\bm\{l\}\_\{<\\mu\}\\right\\\}\\right\)\(2\)×Pr​\(\{Lμ\+1=lμ\+1\}\|\{𝑳<μ\+1=𝒍<μ\+1\}\)×…\\displaystyle\\times\\mathrm\{Pr\}\\left\(\\left\\\{L\_\{\\mu\+1\}=l\_\{\\mu\+1\}\\right\\\}\|\\left\\\{\\bm\{L\}\_\{<\\mu\+1\}=\\bm\{l\}\_\{<\\mu\+1\}\\right\\\}\\right\)\\times\\ldots×Pr​\(\{L\|𝑳\|−1=l\|𝑳\|−1\}\|\{𝑳<\|𝑳\|−1=𝒍<\|𝑳\|−1\}\)\.\\displaystyle\\times\\mathrm\{Pr\}\\left\(\\left\\\{L\_\{\|\\bm\{L\}\|\-1\}=l\_\{\|\\bm\{L\}\|\-1\}\\right\\\}\|\\left\\\{\\bm\{L\}\_\{<\|\\bm\{L\}\|\-1\}=\\bm\{l\}\_\{<\|\\bm\{L\}\|\-1\}\\right\\\}\\right\)\.These conditionals are used to sample \(according to some sampling strategy\) tokens sequentially, so as to produce extended responses𝒍≥μ\\bm\{l\}\_\{\\geq\\mu\}to prompts𝒍<μ\\bm\{l\}\_\{<\\mu\}\.

When an LLM inference process is run locally \(i\.e\. not through a chatbot or a vendor API\) these probabilities are easily available as log \(unnormalized\) probabilities, which can be converted to normalized probability vectors by a softmax operation\. As a consequence we gain access to the LLM’s approximation to the distribution over its training texts, which we may reconstruct according to Equation \([2](https://arxiv.org/html/2605.21726#S3.E2)\)\.

As we demonstrate in the next section, once we have this reconstruction we are no longer constrained to conditional expressions such as those in Equations \([1](https://arxiv.org/html/2605.21726#S3.E1)–[2](https://arxiv.org/html/2605.21726#S3.E2)\), wherein all sequences are complete and contiguous, and the sequences on the left of the conditional bar follow immediately those on the right\. In fact it is possible to construct*arbitrary*expressionsP​r​\(𝑺1\|𝑺2\)Pr\(\\bm\{S\}\_\{1\}\|\\bm\{S\}\_\{2\}\)where𝑺1\\bm\{S\}\_\{1\}and𝑺2\\bm\{S\}\_\{2\}are non\-contiguous, non\-sequential sets of tokens, simply by appropriately prompting an LLM i\.e\. by coercing it to sample the required tokens at the specified sequence locations, and siphoning off the emitted vectors of log\-probability\. We now illustrate this statement by deriving an expression for a score that measures the sensitivity of an LLM response to individual tokens in a prompt\.

### 3\.3Probabilistic Attribution Score

This section describes our approach for attributing importance to prompt tokens in light of the LLM response to the prompt\.

The basic idea is that if we have a prompt𝒑\\bm\{p\}that induces a response𝒓\\bm\{r\}sampled from the distributionPr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑\}\)\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}\\right\\\}\\right\)\(where𝒑\\bm\{p\}and𝒓\\bm\{r\}are contiguous token sequences\), then we would like a measure of attribution of the importance to theμ\\mu\-th prompt tokenpμp\_\{\\mu\}to𝒓\\bm\{r\}given by

Aμ≡log⁡Pr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑\}\)Pr​\(\{𝑹=𝒓\}\|\{𝑷−Pμ=𝒑−pμ\}\),A\_\{\\mu\}\\equiv\\log\\frac\{\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}\\right\\\}\\right\)\}\{\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}\-P\_\{\\mu\}=\\bm\{p\}\-p\_\{\\mu\}\\right\\\}\\right\)\},\(3\)where, again, the notation “\{𝑷−Pμ=𝒑−pμ\}\\left\\\{\\bm\{P\}\-P\_\{\\mu\}=\\bm\{p\}\-p\_\{\\mu\}\\right\\\}” means “the token sequence𝒑\\bm\{p\}omitting the value of tokenpμp\_\{\\mu\}occurring at positionμ\\mu\.” Note that in𝒑−pμ\\bm\{p\}\-p\_\{\\mu\},pμp\_\{\\mu\}itself is not dropped—that is, the sequence𝒑\\bm\{p\}is not shortened by 1—butpμp\_\{\\mu\}could have attained any value in the token vocabulary, instead of being assigned the value given in𝒑\\bm\{p\}\. In effect, the tokenpμp\_\{\\mu\}is masked, in the sense that the information that it contributes to the conditional probability for\{𝑹=𝒓\}\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}is dropped\.

A large value ofAμA\_\{\\mu\}could indicate high importance ofpμp\_\{\\mu\}to𝒓\\bm\{r\}, since it would follow that the inclusion ofpμp\_\{\\mu\}dramatically improves the probability of observing𝒓\\bm\{r\}\. More to the point, the distribution ofAμA\_\{\\mu\}overμ\\muyields an idea of which tokens in the prompt were the chief perpetrators of the response\.

There is a difficulty in computing the expression in Equation \([3](https://arxiv.org/html/2605.21726#S3.E3)\), which is that LLMs do not directly output expressions such asPr​\(\{𝑹=𝒓\}\|\{𝑷−Pμ=𝒑−pμ\}\)\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}\-P\_\{\\mu\}=\\bm\{p\}\-p\_\{\\mu\}\\right\\\}\\right\)\. Instead, they operate sequentially, yielding vectors of categorical probabilities for the next token in a sequence conditioned on all previous tokens\. That is, they supply expressions such as Equation \([1](https://arxiv.org/html/2605.21726#S3.E1)\) or \([2](https://arxiv.org/html/2605.21726#S3.E2)\)\.

The problem, then, is to construct expressions for the denominator of Equation \([3](https://arxiv.org/html/2605.21726#S3.E3)\) using nothing but expressions such as Equation \([1](https://arxiv.org/html/2605.21726#S3.E1)\) or \([2](https://arxiv.org/html/2605.21726#S3.E2)\)\. It turns out that this is indeed possible \[C1, C2, C4\]\.

Per Equation \([3](https://arxiv.org/html/2605.21726#S3.E3)\), we require an expression for the denominatorDD,

D\\displaystyle D≡\\displaystyle\\equivPr​\(\{𝑹=𝒓\}\|\{𝑷−Pμ=𝒑−pμ\}\)\\displaystyle\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}\-P\_\{\\mu\}=\\bm\{p\}\-p\_\{\\mu\}\\right\\\}\\right\)=\\displaystyle=Pr​\(\{𝑹=𝒓\}\|\{𝑷<μ=𝒑<μ\}∩\{𝑷\>μ=𝒑\>μ\}\)\.\\displaystyle\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\right\)\.The required expressions must only contain factors such as those in Equations \([1](https://arxiv.org/html/2605.21726#S3.E1)–[2](https://arxiv.org/html/2605.21726#S3.E2)\) so that they may be read out of the LLM on a side\-channel that reports categorical probabilities over the next token in the sequence\. Assume that the response follows the prompt sequentially, with no gap between them, and that both prompt and response are composed of contiguous sequence locations\. We start with

D\\displaystyle D=\\displaystyle=∑pμ′=0V−1Pr​\(\{𝑹=𝒓\}∩\{Pμ=pμ′\}\|\{𝑷<μ=𝒑<μ\}∩\{𝑷\>μ=𝒑\>μ\}\),\\displaystyle\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\\cap\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\right\),since this expresses the view thatPμP\_\{\\mu\}may realize any value in the vocabulary\. Now, define

𝒑′≡\[p0,…,pμ−1,pμ′,pμ\+1,…,pM−1\]\.\\bm\{p\}^\{\\prime\}\\equiv\\left\[p\_\{0\},\\ldots,p\_\{\\mu\-1\},p\_\{\\mu\}^\{\\prime\},p\_\{\\mu\+1\},\\ldots,p\_\{M\-1\}\\right\]\.Since

\{𝑷<μ=𝒑<μ\}∩\{𝑷\>μ=𝒑\>μ\}∩\{Pμ=pμ′\}=\{𝑷=𝒑′\},\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\cap\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}=\\left\\\{\\bm\{P\}=\\bm\{p\}^\{\\prime\}\\right\\\},we may write

D\\displaystyle D=\\displaystyle=∑pμ′=0V−1Pr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑′\}\)\\displaystyle\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}^\{\\prime\}\\right\\\}\\right\)\(4\)×Pr​\(\{Pμ=pμ′\}\|\{𝑷<μ=𝒑<μ\}∩\{𝑷\>μ=𝒑\>μ\}\)\.\\displaystyle\\times\\mathrm\{Pr\}\\left\(\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\right\)\.The first factor in the summand is in the required form\. As to the second factor, we may apply Bayes’ rule

B​\(pμ′\)\\displaystyle B\(p^\{\\prime\}\_\{\\mu\}\)≡\\displaystyle\\equivPr​\(\{Pμ=pμ′\}\|\{𝑷<μ=𝒑<μ\}∩\{𝑷\>μ=𝒑\>μ\}\)\\displaystyle\\mathrm\{Pr\}\\left\(\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\right\)\(5\)=\\displaystyle=Pr​\(\{𝑷\>μ=𝒑\>μ\}\|\{𝑷≤μ=𝒑≤μ′\}\)×Pr​\(\{Pμ=pμ′\}\|\{𝑷<μ=𝒑<μ\}\)∑pμ′′=0V−1Pr​\(\{𝑷\>μ=𝒑\>μ\}\|\{𝑷≤μ=𝒑≤μ′′\}\)×Pr​\(\{Pμ=pμ′′\}\|\{𝑷<μ=𝒑<μ\}\),\\displaystyle\\frac\{\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{\\leq\\mu\}=\\bm\{p\}^\{\\prime\}\_\{\\leq\\mu\}\\right\\\}\\right\)\\times\\mathrm\{Pr\}\\left\(\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\}\{\\sum\_\{p\_\{\\mu\}^\{\\prime\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{\\leq\\mu\}=\\bm\{p\}^\{\\prime\\prime\}\_\{\\leq\\mu\}\\right\\\}\\right\)\\times\\mathrm\{Pr\}\\left\(\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\},wherein all factors appearing on the right are in the required form\. Combining Equations \([4](https://arxiv.org/html/2605.21726#S3.E4)\) and \([5](https://arxiv.org/html/2605.21726#S3.E5)\),

D\\displaystyle D=\\displaystyle=∑pμ′=0V−1Pr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑′\}\)×B​\(pμ′\)\\displaystyle\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}^\{\\prime\}\\right\\\}\\right\)\\times B\(p^\{\\prime\}\_\{\\mu\}\)
It follows that our attribution measure for prompt locationμ\\muis

Aμ\\displaystyle A\_\{\\mu\}=\\displaystyle=log⁡Pr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑\}\)D\\displaystyle\\log\\frac\{\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}\\right\\\}\\right\)\}\{D\}\(7\)=\\displaystyle=log⁡XY,\\displaystyle\\log\\frac\{X\}\{Y\},where

X\\displaystyle X≡\\displaystyle\\equivPr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑\}\)×∑pμ′=0V−1Pr​\(\{𝑷\>μ=𝒑\>μ\}\|\{𝑷≤μ=𝒑≤μ′\}\)\\displaystyle\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}\\right\\\}\\right\)\\times\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{\\leq\\mu\}=\\bm\{p\}^\{\\prime\}\_\{\\leq\\mu\}\\right\\\}\\right\)\(8\)×Pr​\(\{Pμ=pμ′\}\|\{𝑷<μ=𝒑<μ\}\),\\displaystyle\\hphantom\{\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}\\right\\\}\\right\)\\times\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\}\\times\\mathrm\{Pr\}\\left\(\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\),Y\\displaystyle Y≡\\displaystyle\\equiv∑pμ′=0V−1Pr​\(\{𝑹=𝒓\}\|\{𝑷=𝒑′\}\)×Pr​\(\{𝑷\>μ=𝒑\>μ\}\|\{𝑷≤μ=𝒑≤μ′\}\)\\displaystyle\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}^\{\\prime\}\\right\\\}\\right\)\\times\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{\\leq\\mu\}=\\bm\{p\}^\{\\prime\}\_\{\\leq\\mu\}\\right\\\}\\right\)\(9\)×Pr​\(\{Pμ=pμ′\}\|\{𝑷<μ=𝒑<μ\}\)\.\\displaystyle\\hphantom\{\\sum\_\{p\_\{\\mu\}^\{\\prime\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}=\\bm\{p\}^\{\\prime\}\\right\\\}\\right\)\\times\}\\times\\mathrm\{Pr\}\\left\(\\left\\\{P\_\{\\mu\}=p\_\{\\mu\}^\{\\prime\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\.

### 3\.4Contextual Entropies

A second useful indicator of prompt token sensitivity can be obtained by considering the entropy of each prompt sequence location, conditioned on the remaining prompt tokens, or on the entire remaining context, including the response\.

The entropy of a categorical distribution𝒒\\bm\{q\}with components\[𝒒\]l≡ql≥0\[\\bm\{q\}\]\_\{l\}\\equiv q\_\{l\}\\geq 0,l=0,…,V−1l=0,\\ldots,V\-1satisfying∑l=0V−1ql=1\\sum\_\{l=0\}^\{V\-1\}q\_\{l\}=1is given by the formula

S​\(𝒒\)=−∑l=0V−1ql​log⁡ql\.S\(\\bm\{q\}\)=\-\\sum\_\{l=0\}^\{V\-1\}q\_\{l\}\\log q\_\{l\}\.\(10\)Its significance is that it is an information\-theoretic measure of the concentration of the distribution𝒒\\bm\{q\}over some set of outcomes\. In the case of complete concentration on one outcome,ql=1q\_\{l\}=1,qm≠l=0q\_\{m\\neq l\}=0we haveS​\(𝒒\)=0S\(\\bm\{q\}\)=0, a lower bound, whereas in the case of maximum dispersion∀l:ql=1/V\\forall l:q\_\{l\}=1/Vwe haveS​\(𝒒\)=log⁡VS\(\\bm\{q\}\)=\\log V, an upper bound\([5](https://arxiv.org/html/2605.21726#bib.bib9)\)\.

We set the probabilities𝒒\\bm\{q\}in Equation \([10](https://arxiv.org/html/2605.21726#S3.E10)\) to either the prompt\-conditioned probabilities at sequence locationμ\\mu, i\.e\.

\[𝒒μ\(P\)\]pμ\\displaystyle\\left\[\\bm\{q\}^\{\(P\)\}\_\{\\mu\}\\right\]\_\{p\_\{\\mu\}\}≡\\displaystyle\\equivPr​\(Pμ=pμ\|\{𝑷<μ=𝒑<𝝁\}∩\{𝑷\>μ=𝒑\>μ\}\)\\displaystyle\\mathrm\{Pr\}\\left\(P\_\{\\mu\}=p\_\{\\mu\}\|\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\_\{<\\mu\}\}\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\right\)\(11\)=\\displaystyle=Pr​\(\{𝑷≥μ=𝒑≥𝝁\}\|\{𝑷<μ=𝒑<μ\}\)∑pμ′=0V−1Pr​\(\{𝑷≥μ=𝒑≥𝝁′\}\|\{𝑷<μ=𝒑<μ\}\),\\displaystyle\\frac\{\\mathrm\{Pr\}\\left\(\\\{\\bm\{P\}\_\{\\geq\\mu\}=\\bm\{p\_\{\\geq\\mu\}\}\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\}\{\\sum\_\{p^\{\\prime\}\_\{\\mu\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\left\\\{\\bm\{P\}\_\{\\geq\\mu\}=\\bm\{p\_\{\\geq\\mu\}^\{\\prime\}\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\},or to the prompt\+response\-conditioned probabilities sequence locationμ\\mu, i\.e\.

\[𝒒μ\(P\+R\)\]pμ\\displaystyle\\left\[\\bm\{q\}^\{\(P\+R\)\}\_\{\\mu\}\\right\]\_\{p\_\{\\mu\}\}≡\\displaystyle\\equivPr​\(Pμ=pμ\|\{𝑷<μ=𝒑<𝝁\}∩\{𝑷\>μ=𝒑\>μ\}∩\{𝑹=𝒓\}\)\\displaystyle\\mathrm\{Pr\}\\left\(P\_\{\\mu\}=p\_\{\\mu\}\|\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\_\{<\\mu\}\}\\\}\\cap\\left\\\{\\bm\{P\}\_\{\>\\mu\}=\\bm\{p\}\_\{\>\\mu\}\\right\\\}\\cap\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\\right\)\(12\)=\\displaystyle=Pr​\(\{𝑷≥μ=𝒑≥𝝁\}∩\{𝑹=𝒓\}\|\{𝑷<μ=𝒑<μ\}\)∑pμ′=0V−1Pr​\(\{𝑷≥μ=𝒑≥𝝁′\}∩\{𝑹=𝒓\}\|\{𝑷<μ=𝒑<μ\}\)\.\\displaystyle\\frac\{\\mathrm\{Pr\}\\left\(\\\{\\bm\{P\}\_\{\\geq\\mu\}=\\bm\{p\_\{\\geq\\mu\}\}\\\}\\cap\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\}\{\\sum\_\{p^\{\\prime\}\_\{\\mu\}=0\}^\{V\-1\}\\mathrm\{Pr\}\\left\(\\\{\\bm\{P\}\_\{\\geq\\mu\}=\\bm\{p\_\{\\geq\\mu\}^\{\\prime\}\}\\\}\\cap\\left\\\{\\bm\{R\}=\\bm\{r\}\\right\\\}\|\\left\\\{\\bm\{P\}\_\{<\\mu\}=\\bm\{p\}\_\{<\\mu\}\\right\\\}\\right\)\}\.
Accordingly, the “prompt\-onlycontextual entropy”Sμ\(P\)S^\{\(P\)\}\_\{\\mu\}and “prompt\+responsecontextual entropy”Sμ\(P\+R\)S^\{\(P\+R\)\}\_\{\\mu\}are respectively given by

Sμ\(P\)\\displaystyle S^\{\(P\)\}\_\{\\mu\}≡\\displaystyle\\equivS​\(𝒒μ\(𝑷\)\),\\displaystyle S\\left\(\\bm\{q\}^\{\(\\bm\{P\}\)\}\_\{\\mu\}\\right\),\(13\)Sμ\(P\+R\)\\displaystyle S^\{\(P\+R\)\}\_\{\\mu\}≡\\displaystyle\\equivS​\(𝒒μ\(𝑷\+𝑹\)\)\.\\displaystyle S\\left\(\\bm\{q\}^\{\(\\bm\{P\+R\}\)\}\_\{\\mu\}\\right\)\.\(14\)
In numerical experiments, we occasionally see examples of large changes betweenSμ\(P\)S^\{\(P\)\}\_\{\\mu\}andSμ\(P\+R\)S^\{\(P\+R\)\}\_\{\\mu\}\. It is therefore useful to also construct a divergence measure for prompt token probabilities given the two contexts\. We therefore define the Kullback\-Leibler divergences

KLμ≡KL\(𝒒μ\(P\)\|\|𝒒μ\(P\+R\)\)=∑pμ=0V−1\[𝒒μ\(P\)\]pμlog\[𝒒μ\(P\)\]pμ\[𝒒μ\(P\+R\)\]pμ\.\\mathrm\{KL\_\{\\mu\}\}\\equiv\\mathrm\{KL\}\\left\(\\bm\{q\}^\{\(P\)\}\_\{\\mu\}\|\|\\bm\{q\}^\{\(P\+R\)\}\_\{\\mu\}\\right\)=\\sum\_\{p\_\{\\mu\}=0\}^\{V\-1\}\\left\[\\bm\{q\}^\{\(P\)\}\_\{\\mu\}\\right\]\_\{p\_\{\\mu\}\}\\log\\frac\{\\left\[\\bm\{q\}^\{\(P\)\}\_\{\\mu\}\\right\]\_\{p\_\{\\mu\}\}\}\{\\left\[\\bm\{q\}^\{\(P\+R\)\}\_\{\\mu\}\\right\]\_\{p\_\{\\mu\}\}\}\.\(15\)

## 4Experiments

In this section, we compute attribution scores \(AS\) for77prompts \([Table 1](https://arxiv.org/html/2605.21726#S4.T1)\) and88LLMs \([Table 2](https://arxiv.org/html/2605.21726#S4.T2)\), and evaluate the results\. We currently focus on NLP text generation tasks and use greedy and top\-p\(nucleus, Holtzmanet al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib5)\)decoding\. These choices are for the sake of simplicity only, as our method is not limited to the text generation tasks or the decoding strategy\. Note that these sampling strategies only apply to the initial generation of a response corresponding to a model\. When we computeASorcontextual entropy, we do not sample from the LLM’s conditional probability distributions, but rather coerce the model to include the same tokens from the original response into its context as it generates the log\-probabilities for each new token\.

In each experiment we examine a single response to each prompt from each model\. For top\-p sampling, responses are selected by identifying the most frequent response generated for a given prompt when the model is prompted 500 times\. We chose77prompts from popular LLM benchmarksWanget al\.\([2019b](https://arxiv.org/html/2605.21726#bib.bib10)\); Kwiatkowskiet al\.\([2019](https://arxiv.org/html/2605.21726#bib.bib11)\); Papernoet al\.\([2016](https://arxiv.org/html/2605.21726#bib.bib12)\); Williamset al\.\([2018](https://arxiv.org/html/2605.21726#bib.bib13)\); Wanget al\.\([2019a](https://arxiv.org/html/2605.21726#bib.bib14)\)and modified them while still fulfilling the criteria of the benchmark\. For example, we varied the location of the context \(beginning, spread out, or end of the sentence\), and introduced confusing prompts, or deceptive prompts\.

The full results of these experiments are reported in the supplement\. In this section, we highlight the results that we find most interesting and relevant\.

Table 1:List of PromptsTable 2:Overview of Large Language Models![Refer to caption](https://arxiv.org/html/2605.21726v1/x1.png)Figure 1:Attribution Scores for prompts given the response for the55LLMs with greedy \(left column\) and top\-p decoding \(right column\)\.### 4\.1Attribution Score \(AS\)

[Figure 1](https://arxiv.org/html/2605.21726#S4.F1)shows the Attribution Scores for 5 LLMs and 3 prompts\. The first column of[Figure 1](https://arxiv.org/html/2605.21726#S4.F1)shows greedy and the second shows top\-p decoding schemes\. As we discuss further below, negative values ofAStell us that there are tokens in the LLM vocabulary that the LLM prefers to the token that the user chose during prompting given the LLM response\. In the second row, where the prompt is structured as “Sentence” and “Question”, we see that the LLMs assign high scores to these tokens\. We also note that each model gives a reasonable response\. An exception to this isgpt\-neo, where the default tokenization scheme splits the word “Sentence” into two tokens, and the attribution is not as high as in the case of other LLMs that do not break the word into subwords\. In the first row, the prompt is structured as “Passage” and “Question,” with the answer being a number\. As expected, the responses are varied\. The models do consider the prompt’s structure when devising a response\. Here, each model splits the word signifying the prompt structure, “Passage”, into two subwords, and the attribution score is lower, while the word “Question”, tokenized as\-is, is assigned a high value ofAS\. In row 3, an obviously misleading prompt is used to invoke an LLM response\. With the exception ofLlama, we see that the LLMs using the greedy and top\-p decoding strategies provide incorrect responses\. Note thatgpt\-neoresponds with multiple blank spaces for greedy decoding\.

### 4\.2Contextual Entropy and KL Divergence of Token Distributions Conditioned on the Remaining Context

[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)and[Figure 3](https://arxiv.org/html/2605.21726#S4.F3)showcontextual entropyversusAS, whereASis ordered by value, for77prompts and88LLMs with a response downsampling strategy of greedy and top\-p respectively\. They\-axisshows the contextual entropiesS\(P\)S^\{\(P\)\}andS\(P\+R\)S^\{\(P\+R\)\}of the distribution of each token\. The figures representS\(P\)S^\{\(P\)\}by∘\\circandS\(P\+R\)S^\{\(P\+R\)\}by×\\times\. In each panel, theASaxis is divided up into three ranges: high, near\-zero, and negative, with breaks in the axis between ranges so as to permit convenient representation of the full dynamic range\. The meanS\(P\)S^\{\(P\)\}in each range is indicated by the horizontal dashed lines\. A low contextual entropy means that given the context \(∘\\circor×\\times\) the model prefers fewer replacement tokens for the given token\. These low contextual entropies are mostly concentrated around near\-zeroAS, also seen in near\-zero[Figure 1](https://arxiv.org/html/2605.21726#S4.F1)\(gray\)\. Similarly, high contextual entropy corresponds to more uncertainty in potential token replacements candidates\. These high contextual entropies are mostly concentrated in positive and negativeAS, also seen in blue and red hues in[Figure 1](https://arxiv.org/html/2605.21726#S4.F1)\.

From[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)and[Figure 3](https://arxiv.org/html/2605.21726#S4.F3), near\-zeroAScorresponds to low contextual entropies \(∘\\circ&×\\times\), with a few outliers\. When the∘\\circ&×\\timesoverlap, the context of the response does not change the entropy for a token distribution conditioned on the given context\. In such cases we found that the original prompt token was one of the top90%90\\%candidates for token replacement, further discussed in[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\. In cases when theASis negative, the numerator of[Equation 3](https://arxiv.org/html/2605.21726#S3.E3)is less than the denominator\. This means the model prefers a token other than the current prompt token for the model original response\. However, the original token may still be a part of the top replacement token candidates, further discussed in[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-b\.

![Refer to caption](https://arxiv.org/html/2605.21726v1/x2.png)Figure 2:Contextual Entropy versus Attribution Scores for seven prompts and88LLMs and agreedyresponse\. The x\-axis is stretched from−0\.2\-0\.2to0\.20\.2for clarity\. The entropy is computed with \(×\\times\) and without \(o\) including the response\. The dashed lines indicate the mean of the entropy \(S\(P\)S^\{\(P\)\}\) in the bucket\. The green and red circles are the anomalies discussed in Subsection[4\.2\.2](https://arxiv.org/html/2605.21726#S4.SS2.SSS2)![Refer to caption](https://arxiv.org/html/2605.21726v1/x3.png)Figure 3:Contextual Entropy versus Attribution Scores for six prompts and88LLMs and a response downsampling strategy usingtop\-p\. The x\-axis is stretched from−0\.2\-0\.2to0\.20\.2for clarity\. The entropy is computed with \(×\\times\) and without \(o\) including the response\. The dashed lines indicate the mean of the entropy \(S\(P\)S^\{\(P\)\}\) in the bucket\.#### 4\.2\.1Training Convergence

Thecontextual entropy\(S\(P\)S^\{\(P\)\}andS\(P\+R\)S^\{\(P\+R\)\}\) shows how adding the response to the conditioning information affects the uncertainty in the potential replacements for the token\. Our attribution score together with thecontextual entropyallows us to infer how well the model converges onto the distribution of the training data\. The evidence for this is seen in[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)and[Figure 3](https://arxiv.org/html/2605.21726#S4.F3)\. In the figures, we see that almost always this uncertainty is reduced with the addition of response, and this dip in entropy is larger for higherASvalues\. The entropy is low forASvalues that are in the near\-zero range \(−0\.1\-0\.1to0\.10\.1\), except for older models likegpt2andgpt\-neo\.gpt2is an older English\-only causal LLM\.gpt\-neois newer thangpt2, trained on the Pile,\(Gaoet al\.,[2021](https://arxiv.org/html/2605.21726#bib.bib17); Radfordet al\.,[2019](https://arxiv.org/html/2605.21726#bib.bib18)\)\. Its model card reports better results thangpt2on several language benchmarks, such as LAMBADA\(Zellerset al\.,[2019](https://arxiv.org/html/2605.21726#bib.bib19)\)and HellaSwag\(Papernoet al\.,[2016](https://arxiv.org/html/2605.21726#bib.bib12)\)\. However, both are clearly behind compared to newer models\. In contrast, modern instruct\-tuned models such as Llama 3\.1 Instruct are designed for assistant\-style dialogue, multilingual use, code, and long\-context work\.

The largercontextual entropyat near\-zeroASvalues exhibited by thegpt2andgpt\-neomodels indicate high uncertainty in replacement token choice\. For a model whose training has converged and therefore has a well ascertained distribution over texts, this would be anomalous behavior, because given a “reasonable” prompt \(i\.e\. one consistent with the training data distribution\) a prompt token with near\-zeroASshould correspond to a situation where the identity of that token is implied by the remaining context i\.e\. it can be easily guessed given the rest of the sentence, so that itscontextual entropyshould be low\. If instead such a token has highcontextual entropy, this means that it is a pure noise token, contributing nothing to the probability of the response \(near\-zeroAS\) while being largely unconstrained by the context \(highcontextual entropy\)\. Such noise tokens should be rare for reasonable prompts to a model with good training convergence\. We therefore interpret a high frequency of such noise tokens as indication of poor model training convergence\.

Negative values ofAStell us that there are tokens in the LLM vocabulary that the LLM prefers to the token that the user chose during prompting given the LLM response \[C3\]\. Thecontextual entropyfor negativeASgets higher as we move towards more negative range, indicating that the model finds many other tokens more likely\.

#### 4\.2\.2Investigating Anomalies and Prompt Analysis

![Refer to caption](https://arxiv.org/html/2605.21726v1/x4.png)Figure 4:KL divergences defined in Equation \(15\) for greedy and top\-p sampling\.![Refer to caption](https://arxiv.org/html/2605.21726v1/x5.png)Figure 5:Token probabilities used in the computation ofcontextual entropywith and without the response context\. The prompt tokens are highlighted in the circles\. \(a\) shows the token highlighted with a red oval in[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)and the candidate replacements thellama\-3\.2\-1B\-Instructprefers\. \(b\) shows a token with negativeAS\. \(c\) shows tokens with large\-positiveAS, \(d\) \(e\) shows tokens with nearzeroAS\. Note:G˙\\dot\{G\}represents and empty space\.Figure[4](https://arxiv.org/html/2605.21726#S4.F4)shows the KL divergences between the token distributions with and without the response context, defined in Equation \([15](https://arxiv.org/html/2605.21726#S3.E15)\), for greedy and top\-p sampling\.[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)shows the token probabilities used in the computation ofcontextual entropywith and without the response context, with prompt tokens highlighted in the black circles\.[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(a\) shows the token highlighted with a red oval in[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)and the candidate replacements thellama\-3\.2\-1B\-Instructprefers\.[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(b\) shows a token with negativeAS\.[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(d\) shows the token probabilities with near\-zeroAS\.

[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(d\), where the token probabilities with near\-zeroASare shown, reveals that the candidate replacements for prompt tokens are unchanged even when the response is introduced into the context\. For a negative and positiveAS,[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(b\) and \-\(c\) respectively, the log\-probabilities change for multiple tokens, including the original token, which is reflected in theASscoring as these are the more sensitive tokens in the prompt\.

We now demonstrate how these kinds of plots can be used to detect, investigate, and intervene upon anomalies in LLM output\.

##### Anomaly 1:

As shown in Figure[4](https://arxiv.org/html/2605.21726#S4.F4), the KL divergence is mostly insignificant for near\-zeroASvalues and is higher otherwise\. One exception in the near\-zeroASrange with KL divergence value of12\.1512\.15is in greedy sampling and is highlighted with a red oval in[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)and a red circle in[Figure 4](https://arxiv.org/html/2605.21726#S4.F4)\. Upon further investigation we found that this token, \(the final “??” inWhen was the Cassini launched?\), belongs to prompt 5 \([Table 1](https://arxiv.org/html/2605.21726#S4.T1)\) for thellama\-3\.2\-1B\-Instructmodel\. This is also a token where the prompt\+\+responsecontextual entropyis much lower than prompt\-onlycontextual entropy\(see[Figure 2](https://arxiv.org/html/2605.21726#S4.F2)\) which is unusual at near\-zeroAS\.[Figure 4](https://arxiv.org/html/2605.21726#S4.F4)\(outlined in red\) shows that the model with prompt\+response context has fewer candidate replacements‘:’, ‘?’, ‘ in’, ‘ on’, ‘,’which include the original prompt token,\(?\)\. This exception is also shown in[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(a\), where the model prefers the token “in” over the original prompt token, “?”\. The reason lies in the response, “1997?” Answer: “1997\.” The sentence is a simple declarative sentence with a”\. The models choice of the prompt token, “in”, with response context included makes the sentence more cohesive \(“When was the Cassini launched in 1997?…”\) than with the original prompt, “When was the Cassini launched? 1997?…”\. In effect, the model would have been “happier” with a slight rephrasing of the question\!

##### Anomaly 2:

Another exception is highlighted in green oval in[Figure 2](https://arxiv.org/html/2605.21726#S4.F2), in the near\-zeroASrange\. The anomaly consists of the fact that, unusually,S\(P\+R\)\>S\(P\)S^\{\(P\+R\)\}\>S^\{\(P\)\}for this token\. In effect, the information added by the response over that in the prompt alone has broadened the distribution for this token\. Note that, typically, the token probabilities with near\-zeroAS\(refer to[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(d\)\) reveal that the candidate replacements for prompt tokens remain unchanged even when the response is introduced into the context\.

The anomaly corresponds to prompt 4 with a response: “closely\. He walked over to the door and examined the doorknob more closely\. It was a…”\. The prompt token in question is “police” \(at position 1\)\. Upon further investigation the we found that thellama\-3\.2\-1B\-Instructmodel has a substantially higher preference for the tokens “customs” and “security” than for the original prompt token “police”, refer[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(d\)\. Here, we show tokens whose token probabilities are greater then0\.0050\.005\. This usage aligns more closely with terminology commonly found in US English, as opposed to UK English or other English variants\. Using these examples, we can gain insights into the type of dataset used to train the LLMs and how well the model converges to this training data\.

##### Anomaly 3:

[Figure 5](https://arxiv.org/html/2605.21726#S4.F5)\-\(b\) shows token “age”, from “Passage” at position 1, with negativeAS, belonging to prompt\-5 andllama\-3\.2\-1B\-Instructmodel, refer[Figure 1](https://arxiv.org/html/2605.21726#S4.F1)for greedy sampling\. The other candidate token the model prefers is “entence”, which could be seen as a subword of the word “Sentence\. Upon prompting the model with the word “Passage”, at position 1, replaced with “Sentence”, we were able to get a new, more coherent response: “The answer is 1997\. The sentence is in the past tense, and the subject is…”\. The original response was “1997?” Answer: “1997\.” The sentence is a simple declarative sentence with a”\. Interestingly, when we only replaced the token “age” with “entence” \(giving us “Passentence”\) we obtained the same coherent response\. This example shows that we are able to identify causes of poor output, and engineer prompts to produce improved responses\. At the same time we gain an improved understanding of the nature of the model’s training dataset \[C3\], in that the “Passage/Question” format appears to have been unusual in the dataset, resulting in confused output\.

![Refer to caption](https://arxiv.org/html/2605.21726v1/x6.png)Figure 6:Figure \(a\) shows the entropy of the frequency counts of exact matches in the responses for both greedy and top\-p sampling versusAS\. The frequency counts are computed for each token replacement with the top 90% tokens\. Each circle is one token\. \(b\) shows the percentage of original response for each token versusAS\. In \(a,b\)\-2, the size of the circles indicate the number of top90%90\\%tokens used to replace each token\.

### 4\.3Replacement Entropy of Frequency Counts of the Exact Response Matches

So far, we have computedASfor a single response; here, we will examine howASis related to the preferred responses of an LLM\. For Figure[6](https://arxiv.org/html/2605.21726#S4.F6)\-\(a\) we compute the frequency counts of responses for each token replacement with the top90%90\\%tokens\. To compute the “top90%90\\%tokens,” the LLM output are converted to probabilities, sorted in descending order, and the top tokens are selected whose cumulative probability mass reaches90%90\\%\. First we take the top90%90\\%candidate tokens for each token in the prompt\. Then we prompt the model with the candidate tokens in place of the original token, in each case leaving all the other tokens unchanged\. This results in multiple responses\. We do this for greedy and top\-p sampling strategies\. We then compute the frequency counts of responses \(where all response tokens must match for responses to be counted as the same\)\. Normalized, this frequency distribution becomes an empirical probability distribution\. We compute the entropy of this distribution, and refer to it as thereplacement entropy\. We also tally the number of exact matches to the original response, for both greedy and top\-p sampling strategies\.

Figure[6](https://arxiv.org/html/2605.21726#S4.F6)\-\(a\), showsreplacement entropyversusAS\. The size of the circles on the plots on the right indicates the number of top 90% tokens used to replace each token \(each circle is one token\) for both greedy and top\-p strategies\. As theASincreases \(or goes more negative\) the variability in responses increase even when there are fewer candidate replacement tokens\. WhenASis near zero, there are fewer top 90% replacement tokens and relatively lowerreplacement entropy\. Figure[6](https://arxiv.org/html/2605.21726#S4.F6)\-\(b\) shows the percentage of original response for each token versusAS\. The large the value ofASthe lower is the original response count even with larger number of candidate replacements when compared to near\-zeroAS\. We can conclude that the highAStokens are more sensitive, have more variation in the responses, and do not prefer the original response even when the top90%90\\%candidate tokens are chosen\. Unlikecontextual entropy, which computes the entropy from log\-probabilities extracted from the models,replacement entropyuses the frequency counts of the responses and we can see a clear correlation betweenASand the LLM response across multiple models and prompts \[C1, C3\]\.

## 5Evaluation

![Refer to caption](https://arxiv.org/html/2605.21726v1/x7.png)Figure 7:XAI Evaluation Metrics for66models and77prompts for greedy sampling\. Model11= gemma\-3\-1b\-it,22= gpt2,33= gpt\-neo\-1\.3B,44= Llama\-3\.2\-1B\-Instruct,77= Qwen2\.5\-1\.5B\-Instruct, and88= Qwen3\-1\.7B\.![Refer to caption](https://arxiv.org/html/2605.21726v1/x8.png)Figure 8:XAI Evaluation Metrics for66models and77prompts for top\-p sampling\. Model11= gemma\-3\-1b\-it,22= gpt2,33= gpt\-neo\-1\.3B,44= Llama\-3\.2\-1B\-Instruct,77= Qwen2\.5\-1\.5B\-Instruct, and88= Qwen3\-1\.7B\.We evaluate our token\-attribution methods across66LLMs, given a prompt,xx, and a fixed response,yy\. We test whether the attribution scores are faithful to the model’s actual behavior and compare it with other XAI methods:Input\_X\_Gradient\(Shrikumaret al\.,[2017](https://arxiv.org/html/2605.21726#bib.bib33)\),Gradient\_Shapley\(Lundberg and Lee,[2017](https://arxiv.org/html/2605.21726#bib.bib24)\),Lime\(Ribeiroet al\.,[2016a](https://arxiv.org/html/2605.21726#bib.bib34)\),Occlusion\(Zeiler and Fergus,[2014](https://arxiv.org/html/2605.21726#bib.bib35)\), andReAgent,Zhao and Shan \([2024](https://arxiv.org/html/2605.21726#bib.bib36)\)\.

Input\_X\_Gradient\(i​n​p​u​t​\_​x​\_​g​r​a​d​i​e​n​tinput\\\_x\\\_gradient\) is a local, first\-order attribution technique that assigns token\-level importance by taking the element\-wise product of the input representation \(typically token embeddings\) and the gradient of a chosen target quantity \(e\.g\., a log\-probability\) with respect to that input\. This yields a saliency score that approximates the token’s marginal influence under an infinitesimal perturbation assumption\.Gradient\_Shapley\(often implemented via Shapley\-inspired gradient methods such asg​r​a​d​i​e​n​t​\_​s​h​a​pgradient\\\_shap\) instead estimates token contributions by aggregating gradient\-based attributions across multiple interpolations between a baseline and the observed input, thereby approximating Shapley\-style credit assignment and typically reducing sensitivity to the local gradient at the input point\.Occlusionis a perturbation\-based attribution method that quantifies token importance by systematically masking \(or replacing\) each token or token span and measuring the induced change in a specified target score, such as the log\-probability of a generated token\. By comparing these score differences to the baseline, it estimates each input region’s contribution\. Occlusion works without access to internal gradients\.LIME\(Local Interpretable Model\-agnostic Explanations\) is another perturbation\-based method that samples many locally perturbed variants of an input \(e\.g\., by deleting or masking tokens\), but it fits a simple, interpretable surrogate model to approximate the black\-box model’s behavior in that neighborhood\. The surrogate’s coefficients are then interpreted as token\-level attributions, providing a locally faithful explanation of which input components most influence the target prediction\.ReAgentis a perturbation\-based, model\-agnostic method for autoregressive generation that scores each context token by replacing it with plausible alternatives and measuring the resulting change in a target quantity \(e\.g\., the next\-token log\-probability\)\. By repeating these replacement probes \(replacing\_ratio and num\_probes\) and aggregating the induced score drops, it identifies the minimal subset of tokens that most strongly sustains the model’s prediction\.

We apply44evaluation methods to test the faithfulness of the XAI methods to the model’s actual behavior\. The evaluation methods used are Infidelity\(Yehet al\.,[2019](https://arxiv.org/html/2605.21726#bib.bib21)\), Deletion/NAOPC curves\(Edinet al\.,[2025](https://arxiv.org/html/2605.21726#bib.bib22)\), Comprehensiveness\(DeYounget al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib27)\), and Sufficiency\(DeYounget al\.,[2020](https://arxiv.org/html/2605.21726#bib.bib27)\)\. Infidelity measures how well an explanation’s attributions predict the actual change in a chosen modelffwhen the input is randomly perturbed\. For each perturbation,x′x^\{\\prime\}, we compare the score changef​\(x\)−f​\(x′\)f\(x\)\-f\(x^\{\\prime\}\)to the change the explanation predicts \(often a dot product between attributions and the perturbation mask\), then average the squared error across many perturbations\. Lower infidelity means the explanation is more faithful to the model under that perturbation distribution, where0would be a perfect prediction\. The absolute magnitude depends on the scale offfand the perturbation strength, so it’s most interpretable when comparing methods under the same setup\. Deletion evaluates faithfulness by ranking tokens by attribution and progressively “removing” or neutralizing the top\-k tokens, then recomputing the model scoreff\(oftenl​o​g​\(P​\(y\|x\)\)log\(P\(y\|x\)\)\) for a fixed response\)\. A good explanation causes the score to drop quickly when its highest\-ranked tokens are deleted\. NAOPC \(Area Over the Perturbation Curve\) summarizes the score as the average dropf​\(x\)−f​\(x​\(k\)\)f\(x\)\-f\(x\(k\)\)across deletion steps, and a higher NAOPC indicates that the explainer’s top tokens are more causally important for the model’s behavior\. Results depend on the deletion operator \(e\.g\., masking vs replacement\) and should be compared under identical perturbations\. Here, we use replacement\. Comprehensiveness measures whether the tokens the explanation marks as important are truly necessary for the model’s decision\. We take the top\-k tokensTkT\_\{k\}and measure how much the model score falls when we remove them:f​\(x\)−f​\(x/Tk\)f\(x\)\-f\(x/T\_\{k\}\)\. Higher comprehensiveness means this removal substantially harms the model score, suggesting the explanation captured key evidence\. Because token deletion can introduce a distribution shift, many implementations replace tokens with a baseline to keep the length or tokenization stable\. Sufficiency asks whether the selected top\-k tokens are nearly enough on their own to support the model’s score\. We retainTkT\_\{k\}and compute the drop:f​\(x\)−f​\(Tk\)f\(x\)\-f\(T\_\{k\}\)\. Lower sufficiency \(a small drop\) indicates the explanation’s subset retains most of the model’s evidence for the output, while a large drop suggests the subset is incomplete\. Sufficiency is often reported alongside comprehensiveness to distinguish “necessary” evidence from “nearly sufficient” evidence\.

We use the Inseq library\(Sartiet al\.,[2023](https://arxiv.org/html/2605.21726#bib.bib20)\), a unified, research\-oriented framework for computing and visualizing token\-level attributions for modern sequence models, to compute the attribution scores of other XAI methods\. We then compareASwith other XAI methods using the evaluation metrics\. This approach improves comparability and reproducibility of XAI analyses across models, attribution techniques, and decoding settings, thereby facilitating systematic evaluation and reporting\. We compareASagainst five XAI methods:Input\_X\_Gradient,Gradient\_Shapley,Lime,Occlusion, andReAgent\.

Figure[7](https://arxiv.org/html/2605.21726#S5.F7)and Figure[8](https://arxiv.org/html/2605.21726#S5.F8)show the evaluation metrics for66LLMs and77prompts for greedy and top\-p sampling strategies, respectively\. The direction of the arrow indicates which values are better; for example, up means higher is better, and down means lower is better\. In most cases, we see thatASperforms either better or within a close range of the other more popular methods\. Also, note that whileASis model\-agnostic and provides results across multiple LLMs,i​n​p​u​t​\_​x​\_​g​r​a​d​i​e​n​tinput\\\_x\\\_gradientandg​r​a​d​i​e​n​t​\_​s​h​a​p​l​e​ygradient\\\_shapleydepend on the model architecture\. Note that theOLMomodels are not implemented in the current version of Inseq and are not shown in the figures, even though we can computeASfor these models\. As of writing the paper, Inseq could not applyReAgentto theLlamamodel in our system\. All of the XAI implementations require a user\-defined set of parameters for computing the attribution scores\. For example, ReAgent hasr​e​p​l​a​c​e​m​e​n​t​\_​r​a​t​i​oreplacement\\\_ratioandn​u​m​\_​p​r​o​b​e​snum\\\_probes\. Hence, the final token attributions depend on the user\-defined parameters\. By contrast,ASis a parameter\-free, model\-agnostic attribution method that provides attribution based only on the model’s log probabilities for a given prompt and response\. With sensitivity built intoASand withASbeing parameter\-free, our method could thus serve as a baseline for hyperparameter selection for other XAI methods \[C1, C2, C4\]\.

![Refer to caption](https://arxiv.org/html/2605.21726v1/x9.png)Figure 9:Compute time forASfor greedy and nucleus \(top\-p\) decoding across multiple LLMs\. Top row shows the total GPU hours and bottom row shows GPU seconds for each token replacement from EquationLABEL:eq:Denom\. The red text on each stacked bar is the total tokens in the model vocabulary\. The prompt token length is the length of the LLM tokenized prompt\.[Figure 9](https://arxiv.org/html/2605.21726#S5.F9)shows the compute time forASfor greedy decoding across multiple LLMs\. We show total GPU hours on the top and GPU seconds for each token replacement from EquationLABEL:eq:Denomat the bottom\. The red text on each stacked bar is the total tokens in the model vocabulary\. As the model vocabulary increases, the time to compute theASalso increases\. We use an A100 GPU of a leadership\-class supercomputer to compute the results\. The goal of this work is to gain a deeper understanding of how large language models produce responses, using a model\-agnostic attribution scoring method that makes minimal assumptions\. This approach comes at a cost of higher GPU hours\. Note, however, that it is likely that theASmethod can be made more computationally\-efficient than it is in our current implementation, by restricting the summations over replacement tokens in Equations \([8](https://arxiv.org/html/2605.21726#S3.E8)–[9](https://arxiv.org/html/2605.21726#S3.E9)\) to only the top probability tokens\. We will investigate such potential computational savings in future work\.

Multiple repeated prompting cause log\-probability variation on GPUs, even when the generated text is identical for the same prompt, and this is typically attributable to numerical and algorithmic nondeterminism in accelerated inference\. In particular, CUDA/cuDNN kernels and fused attention implementations may employ parallel reductions, atomics, or autotuned kernel selection, yielding small floating\-point differences because arithmetic is not strictly associative\. These effects are often amplified when precision is reduced \(FP16/BF16\) or when matrix multiplications are TF32\-enabled\. Additional contributors include stochastic decoding configurations \(e\.g\., sampling\) that may, by coincidence, produce the same token sequence, and inadvertent training\-mode behavior \(e\.g\., dropout\) if the model is not set to evaluation/inference mode\. Consequently, the token choice can remain stable while the underlying log\-probabilities shift slightly, unless determinism is enforced via fixed decoding, deterministic algorithm settings, disabled benchmarking/autotuning, and stricter precision controls, which is precisely what we do forAScomputations, since we use GPUs for speedy calculations\. TheAScomputations would be expedited if GPU non\-determinism did not need to be considered\.

In[section 4](https://arxiv.org/html/2605.21726#S4), we have seen empirical results of promising near\-zero AS behavior, which hints towards the quality of model convergence\. A part of our future work is to use this finding to identify model checkpoints to guide the LLM training\. Our hypothesis is that this finding could help identify bad training and insufficient model capacity during the training process\.

## 6Conclusion

We have developed an attribution scoring method,AS, that leverages the generative nature of current Large Language Models \(LLMs\), expressed by the conditional probabilities they compute to sample each response token given the previous tokens\. Our model\-agnostic probabilistic token attribution measure exploits the connection between LLMs and stochastic processes, situating all LLMs in the context of the well\-developed mathematical theory of stochastic processes, and permitting broad cross\-comparisons of LLM output across all such models\. We use this framework to reconstruct the probability distribution over texts that LLMs build up during training from the log\-probabilities emitted by the models during inference\. From our experiments and evaluations usingcontextual entropy,replacement entropy, and KL divergence of token distributions conditioned on the entire remaining context, we can draw conclusions about LLM stability\. We found that GPT \-2 and GPT\-Neo are less effective at text generation than newer models like Llama\. Despite having no access to the training data, we can draw conclusions about how well the models converge on the training data distribution\. We were also able to tweak the prompts with modified tokens to generate a desired response\. AS has sensitivity built into its definition and is parameter\-free, which could serve as a guide to selecting hyperparameters for other XAI and XAI evaluation methods that are faster compared to our approach\. We evaluate 8 models across 7 prompts and identify token sensitivity and response stability, thereby improving interpretability and guiding users to focus on uncertain or unstable parts of the generation\.

Acknowledgments and Disclosure of Funding

This research used resources of the Argonne Leadership Computing Facility, which is a U\.S\. Department of Energy Office of Science User Facility operated under contract DE\-AC02\-06CH11357\. All authors were supported by the Office of Science, U\.S\. Department of Energy, under contract DE\-AC02\-06CH11357\.

## References

- A\. Barredo Arrieta, N\. Díaz\-Rodríguez, J\. Del Ser, A\. Bennetot, S\. Tabik, A\. Barbado, S\. Garcia, S\. Gil\-Lopez, D\. Molina, R\. Benjamins, R\. Chatila, and F\. Herrera \(2020\)Explainable artificial intelligence \(xai\): concepts, taxonomies, opportunities and challenges toward responsible ai\.Information Fusion58,pp\. 82–115\.External Links:ISSN 1566\-2535,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.inffus.2019.12.012),[Link](https://www.sciencedirect.com/science/article/pii/S1566253519308103)Cited by:[§1](https://arxiv.org/html/2605.21726#S1.p3.1)\.
- Probability and measure\.Wiley Series in Probability and Statistics,Wiley\.External Links:ISBN 9780471007104,LCCN gb95051456Cited by:[§3\.1](https://arxiv.org/html/2605.21726#S3.SS1.p5.19),[§3](https://arxiv.org/html/2605.21726#S3.p3.12)\.
- J\. DeYoung, S\. Jain, N\. F\. Rajani, E\. Lehman, C\. Xiong, R\. Socher, and B\. C\. Wallace \(2020\)ERASER: a benchmark to evaluate rationalized NLP models\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4443–4458\.Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p3.1),[§2](https://arxiv.org/html/2605.21726#S2.p6.1),[§5](https://arxiv.org/html/2605.21726#S5.p3.13)\.
- J\. Edin, A\. G\. Motzfeldt, C\. L\. Christensen, T\. Ruotsalo, L\. Maaløe, and M\. Maistro \(2025\)Normalized AOPC: fixing misleading faithfulness metrics for feature attributions explainability\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 1715–1730\.External Links:[Link](https://aclanthology.org/2025.acl-long.86/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.86),ISBN 979\-8\-89176\-251\-0Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p3.13)\.
- \[5\]\(2005\)Elements of information theory\.John Wiley & Sons, Ltd\.External Links:ISBN 9780471748823,[Document](https://dx.doi.org/https%3A//doi.org/10.1002/047174882X.fmatter)Cited by:[§3\.4](https://arxiv.org/html/2605.21726#S3.SS4.p2.10)\.
- N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah \(2021\)A mathematical framework for transformer circuits\.Note:Transformer Circuits ThreadExternal Links:[Link](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p1.1),[§2](https://arxiv.org/html/2605.21726#S2.p5.1)\.
- L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy \(2021\)The pile: an 800gb dataset of diverse text for language modeling\.CoRRabs/2101\.00027\.External Links:[Link](https://arxiv.org/abs/2101.00027),2101\.00027Cited by:[§4\.2\.1](https://arxiv.org/html/2605.21726#S4.SS2.SSS1.p1.4)\.
- A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi \(2020\)The curious case of neural text degeneration\.External Links:1904\.09751,[Link](https://arxiv.org/abs/1904.09751)Cited by:[§4](https://arxiv.org/html/2605.21726#S4.p1.2)\.
- S\. Jain and B\. C\. Wallace \(2019\)Attention is not explanation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 3543–3556\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1357)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p4.1)\.
- N\. Kokhlikyan, V\. Miglani, M\. Martin, E\. Wang, B\. Alsallakh, J\. Reynolds, A\. Melnikov, N\. Kliushkina, C\. Araya, S\. Yan, and O\. Reblitz\-Richardson \(2020\)Captum: a unified and generic model interpretability library for PyTorch\.External Links:2009\.07896,[Document](https://dx.doi.org/10.48550/arXiv.2009.07896)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p2.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov \(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276),[Link](https://doi.org/10.1162/tacl_a_00276),https://direct\.mit\.edu/tacl/article\-pdf/doi/10\.1162/tacl\_a\_00276/1923288/tacl\_a\_00276\.pdfCited by:[§4](https://arxiv.org/html/2605.21726#S4.p2.1)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 4765–4774\.Cited by:[§1](https://arxiv.org/html/2605.21726#S1.p3.1),[§2](https://arxiv.org/html/2605.21726#S2.p2.1),[§5](https://arxiv.org/html/2605.21726#S5.p1.3)\.
- H\. Luo and L\. Specia \(2024\)From understanding to utilization: a survey on explainability for large language models\.ArXivabs/2401\.12874\.External Links:[Link](https://api.semanticscholar.org/CorpusID:267095032)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p1.1),[§2](https://arxiv.org/html/2605.21726#S2.p3.1),[§2](https://arxiv.org/html/2605.21726#S2.p6.1)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández \(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1525–1534\.External Links:[Link](https://aclanthology.org/P16-1144/),[Document](https://dx.doi.org/10.18653/v1/P16-1144)Cited by:[§4\.2\.1](https://arxiv.org/html/2605.21726#S4.SS2.SSS1.p1.4),[§4](https://arxiv.org/html/2605.21726#S4.p2.1)\.
- P\. J\. Phillips, C\. Hahn, P\. Fontana, A\. Yates, K\. K\. Greene, D\. Broniatowski, and M\. A\. Przybocki \(2021\)Four principles of explainable artificial intelligence\.NIST Interagency/Internal Report \(NISTIR\), National Institute of Standards and Technology, Gaithersburg, MD\(en\)\.External Links:[Link](https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=933399),[Document](https://dx.doi.org/https%3A//doi.org/10.6028/NIST.IR.8312)Cited by:[§1](https://arxiv.org/html/2605.21726#S1.p2.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§4\.2\.1](https://arxiv.org/html/2605.21726#S4.SS2.SSS1.p1.4)\.
- M\. T\. Ribeiro, S\. Singh, and C\. Guestrin \(2016a\)“Why should i trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ’16,New York, NY, USA,pp\. 1135–1144\.External Links:ISBN 9781450342322,[Link](https://doi.org/10.1145/2939672.2939778),[Document](https://dx.doi.org/10.1145/2939672.2939778)Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p1.3)\.
- M\. T\. Ribeiro, S\. Singh, and C\. Guestrin \(2016b\)“Why should i trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1135–1144\.External Links:[Document](https://dx.doi.org/10.1145/2939672.2939778)Cited by:[§1](https://arxiv.org/html/2605.21726#S1.p3.1),[§2](https://arxiv.org/html/2605.21726#S2.p2.1)\.
- A\. Rogers, O\. Kovaleva, and A\. Rumshisky \(2020\)A primer in BERTology: what we know about how BERT works\.Transactions of the Association for Computational Linguistics8,pp\. 842–866\.External Links:[Link](https://aclanthology.org/2020.tacl-1.54/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00349)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p1.1),[§2](https://arxiv.org/html/2605.21726#S2.p5.1)\.
- C\. Rudin \(2019\)Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead\.Nat\. Mach\. Intell\.1\(5\),pp\. 206–215\(en\)\.Cited by:[§1](https://arxiv.org/html/2605.21726#S1.p2.1)\.
- G\. Sarti, N\. Feldhus, L\. Sickert, O\. van der Wal, M\. Nissim, and A\. Bisazza \(2023\)Inseq: an interpretability toolkit for sequence generation models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),Toronto, Canada,pp\. 421–435\.External Links:[Link](https://aclanthology.org/2023.acl-demo.40),[Document](https://dx.doi.org/10.18653/v1/2023.acl-demo.40)Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p4.1)\.
- A\. Shrikumar, P\. Greenside, and A\. Kundaje \(2017\)Learning important features through propagating activation differences\.InProceedings of the 34th International Conference on Machine Learning \- Volume 70,ICML’17,pp\. 3145–3153\.Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p1.3)\.
- S\. Sinha, H\. Chen, A\. Sekhon, Y\. Ji, and Y\. Qi \(2021\)Perturbing inputs for fragile interpretations in deep natural language processing\.InProceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,J\. Bastings, Y\. Belinkov, E\. Dupoux, M\. Giulianelli, D\. Hupkes, Y\. Pinter, and H\. Sajjad \(Eds\.\),Punta Cana, Dominican Republic,pp\. 420–434\.External Links:[Link](https://aclanthology.org/2021.blackboxnlp-1.33/),[Document](https://dx.doi.org/10.18653/v1/2021.blackboxnlp-1.33)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p2.1)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017\)Axiomatic attribution for deep networks\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 3319–3328\.Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p2.1)\.
- A\. Wang, Y\. Pruksachatkun, N\. Nangia, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman \(2019a\)SuperGLUE: a stickier benchmark for general\-purpose language understanding systems\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2605.21726#S4.p2.1)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman \(2019b\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.External Links:1804\.07461,[Link](https://arxiv.org/abs/1804.07461)Cited by:[§4](https://arxiv.org/html/2605.21726#S4.p2.1)\.
- S\. Wiegreffe and Y\. Pinter \(2019\)Attention is not not explanation\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 11–20\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1002)Cited by:[§2](https://arxiv.org/html/2605.21726#S2.p4.1)\.
- A\. Williams, N\. Nangia, and S\. R\. Bowman \(2018\)A broad\-coverage challenge corpus for sentence understanding through inference\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 1112–1122\.External Links:[Link](https://aclanthology.org/N18-1101/),[Document](https://dx.doi.org/10.18653/v1/N18-1101)Cited by:[§4](https://arxiv.org/html/2605.21726#S4.p2.1)\.
- C\. Yeh, C\. Hsieh, A\. S\. Suggala, D\. I\. Inouye, and P\. Ravikumar \(2019\)On the \(in\)fidelity and sensitivity of explanations\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p3.13)\.
- M\. D\. Zeiler and R\. Fergus \(2014\)Visualizing and understanding convolutional networks\.InComputer Vision – ECCV 2014,D\. Fleet, T\. Pajdla, B\. Schiele, and T\. Tuytelaars \(Eds\.\),Cham,pp\. 818–833\.External Links:ISBN 978\-3\-319\-10590\-1Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p1.3)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.CoRRabs/1905\.07830\.External Links:[Link](http://arxiv.org/abs/1905.07830),1905\.07830Cited by:[§4\.2\.1](https://arxiv.org/html/2605.21726#S4.SS2.SSS1.p1.4)\.
- Z\. Zhao and B\. Shan \(2024\)ReAGent: a model\-agnostic feature attribution method for generative language models\.External Links:2402\.00794,[Link](https://arxiv.org/abs/2402.00794)Cited by:[§5](https://arxiv.org/html/2605.21726#S5.p1.3)\.

## Appendix A

In this appendix, we provide the Attribution score for 7 prompts and 8 models for both greedy and topp sampling\.

![Refer to caption](https://arxiv.org/html/2605.21726v1/x10.png)Figure 10:Attribution Scores for prompts given the response for the88LLMs and44prompts with greedy decoding\.![Refer to caption](https://arxiv.org/html/2605.21726v1/x11.png)Figure 11:Attribution Scores for prompts given the response for the88LLMs and33prompts with greedy decoding\.![Refer to caption](https://arxiv.org/html/2605.21726v1/x12.png)Figure 12:Attribution Scores for prompts given the response for the88LLMs and44prompts with top\-p decoding\.![Refer to caption](https://arxiv.org/html/2605.21726v1/x13.png)Figure 13:Attribution Scores for prompts given the response for the88LLMs and33prompts with top\-p decoding\.