Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions

# Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions
Source: [https://arxiv.org/html/2608.18539](https://arxiv.org/html/2608.18539)
Ruiyang QinQingzhuo WangAffiliation:School of Computer Science and Technology, Tongji University, Shanghai, ChinaTian WangAffiliation:School of Computer Science and Technology, Tongji University, Shanghai, ChinaZhihua WeiAffiliation:School of Computer Science and Technology, Tongji University, Shanghai, ChinaWen ShenAffiliation:School of Computer Science and Technology, Tongji University, Shanghai, ChinaCorrespondence to:[wenshen@tongji\.edu\.cn](mailto:)

###### Abstract

The remarkable capabilities of large language models \(LLMs\) are often undermined by their instability\. Even subtle and semantically irrelevant changes in prompts can cause dramatic fluctuations in performance, a phenomenon known as prompt sensitivity\. Previous studies typically evaluate prompt sensitivity by comparing the LLM’s final outputs when prompts change\. However, such coarse\-grained metrics fail to explain the internal reasons for prompt sensitivity\. In this paper, we introduce interactions as a fine\-grained tool to analyze prompt sensitivity of LLMs\. Specifically, we decompose the output score of the LLM into a set of interactions\. Each interaction represents a nonlinear relationship involving a set of input variables\. We discover that subtle changes to prompts can trigger severe instability in interactions, even when the outputs of the LLM remain the same\. To this end, we propose an Interaction\-based Prompt Sensitivity \(IPS\) metric by quantifying changes in interactions when we introduce subtle changes to prompts\. We apply the IPS metric to 50 open\-source LLMs and uncover four factors that reduce the prompt sensitivity of LLMs, including supervised fine\-tuning, increased model scales, dense architectures, and few\-shot learning\. More crucially, we discover a common mechanism by which these four factors reduce prompt sensitivity: all four factors tend to reduce the prompt sensitivity of low\-order interactions \(i\.e\., interactions involving few input variables\)\.

## 1Introduction

LLMs have demonstrated exceptional proficiency in numerous natural language processing tasks\([43](https://arxiv.org/html/2608.18539#bib.bib2);[55](https://arxiv.org/html/2608.18539#bib.bib1);[3](https://arxiv.org/html/2608.18539#bib.bib3);[9](https://arxiv.org/html/2608.18539#bib.bib4)\), a success largely driven by the effectiveness of prompting\. However, this power is undermined by prompt sensitivity\. That is, semantically unimportant changes to the prompt can result in divergent outputs\([41](https://arxiv.org/html/2608.18539#bib.bib5)\)\. Current research\([41](https://arxiv.org/html/2608.18539#bib.bib5);[10](https://arxiv.org/html/2608.18539#bib.bib8);[29](https://arxiv.org/html/2608.18539#bib.bib9);[1](https://arxiv.org/html/2608.18539#bib.bib6);[18](https://arxiv.org/html/2608.18539#bib.bib7);[35](https://arxiv.org/html/2608.18539#bib.bib10)\)on evaluating prompt sensitivity only focuses on the LLM’s final output\. These output\-based metrics typically measure changes in performance, such as task accuracy or output consistency\. As coarse\-grained measures, these metrics only reveal the consequences of prompt sensitivity \(e\.g\., the LLM’s prediction changes from one answer to another one\) but fail to explain its underlying reasons\.

In this paper, we aim to evaluate the prompt sensitivity of LLMs from a fine\-grained perspective and investigate the underlying reasons why certain factors can decrease the prompt sensitivity of LLMs\. Recent research\([11](https://arxiv.org/html/2608.18539#bib.bib20);[54](https://arxiv.org/html/2608.18539#bib.bib23);[37](https://arxiv.org/html/2608.18539#bib.bib25);[40](https://arxiv.org/html/2608.18539#bib.bib26)\)has utilized interactions to explain the fine\-grained inference logic of deep neural networks \(DNNs\)\. Inspired by these studies, we introduce the interactions framework to fine\-grainedly analyze the prompt sensitivity of LLMs\.

Specifically, given a sentence𝒙\\bm\{x\}withnninput variables \(e\.g\., words or tokens\) indexed byN=\{1,2,…,n\}N=\\\{1,2,\\ldots,n\\\}, an interaction represents an intricate nonlinear relationship associated with a specific combination of input variables\. Consider the sentence𝒙=“He is a green hand\.”\\bm\{x\}=\\textit\{\`\`He\\ is\\ a\\ green\\ hand\."\}In this context, the idiom “green hand” carries the meaning of “beginner”\. The joint presence of the input variables in the setS=\{green,hand\}⊆NS=\\\{\\textit\{green\},\\textit\{hand\}\\\}\\\!\\subseteq\\\!Ntriggers a special interaction effect\. This interaction effect, denoted asISI\_\{S\}, pushes the network’s inference towards the semantic meaning of“beginner\.”[27](https://arxiv.org/html/2608.18539#bib.bib17)have mathematically demonstrated that the scalar outputv⁡\(𝒙\)v\(\\bm\{x\}\)of a DNN is always equivalent to the output of an interaction\-based logical modelϕ⁡\(𝒙\)=∑S⊆NIS\\phi\(\\bm\{x\}\)=\\sum\\nolimits\_\{S\\subseteq N\}I\_\{S\}\. That is,v⁡\(𝒙\)=ϕ⁡\(𝒙\)=∑S⊆NISv\(\\bm\{x\}\)=\\phi\(\\bm\{x\}\)=\\sum\\nolimits\_\{S\\subseteq N\}I\_\{S\}\. Thus, the inference logic of a DNN can be explained by a set of interactions\.

Figure 1:Using interactions for fine\-grained analysis of prompt sensitivity\. Given an input𝒙\\bm\{x\}and a prompt templateTT, the LLM’s output scorev⁡\(\`​\`​B​"\|𝒙,T\)v\(\`\`B"\|\\bm\{x\},T\)is equivalent to the output of an interaction\-based logical modelϕ⁡\(𝒙\)\\phi\(\\bm\{x\}\), i\.e\.,v⁡\(\`​\`​B​"\|𝒙,T\)=ϕ⁡\(𝒙\)=∑S⊆NISv\(\`\`B"\|\\bm\{x\},T\)=\\phi\(\\bm\{x\}\)=\\sum\\nolimits\_\{S\\subseteq N\}I\_\{S\}\. In this way, we can uncover the underlying reasons for the prompt sensitivity of LLMs by analyzing detailed interaction patterns\. Specifically, we divide interactions into those that are stable to prompt changes \(i\.e\.stable interaction\) and those that are not \(i\.e\.unstable interaction\)\.Coarse\-grained analysis vs\. fine\-grained analysis\.Traditional analysis of prompt sensitivity can only coarsely reflect whether the LLM’s prediction alters when the prompt changes\. Beyond this, we introduce an interaction\-based logical modelϕ⁡\(𝒙\)\\phi\(\\bm\{x\}\)as a fine\-grained analytical tool to analyze the stable and unstable interactions encoded by the LLM for each specific sample\. As shown in Figure[1](https://arxiv.org/html/2608.18539#S1.F1), when we keep the same input𝒙\\bm\{x\}but introduce semantically equivalent alterations to the prompt templateTT\(e\.g\., change “Answers” to “ANSWERS”\), traditional coarse\-grained analysis can only reveal that the LLM’s prediction changes from the correct answer “B” to the incorrect answer “A” but fails to explain why or how this change occurs\. In contrast, our interaction\-based analysis precisely identifies unstable interactions that may contribute to the prompt sensitivity of the LLM\. An interaction is considered stable if its effect changes minimally \(e\.g\.,I\{lay, eggs\}I\_\{\\\{\\textit\{lay, eggs\}\\\}\}shifts from 0\.6242 to 0\.6076\)\. Conversely, an interaction is deemed unstable if its effect fluctuates dramatically \(e\.g\.,I\{NOT, lay, eggs, dogs\}I\_\{\\\{\\textit\{NOT, lay, eggs, dogs\}\\\}\}shifts from 0\.5175 to \-0\.0594\)\. Strikingly, we find that unstable interactions exist even when the LLM’s final output remains the same\. This indicates that our fine\-grained analysis reveals potential instability that is entirely invisible to traditional, output\-level metrics\.

Building on these findings, we leverage the interaction framework to propose a fine\-grained metric to evaluate the prompt sensitivity of LLMs, which is termedInteraction\-basedPromptSensitivity \(IPS\)\. This metric quantifies changes in interaction patterns when LLMs process different prompts\. We apply the proposed IPS metric to evaluate the prompt sensitivity of 50 open\-source LLMs\. However, drawing conclusions about which LLM families are more or less sensitive from this ranking is challenging, as an LLM’s prompt sensitivity stems from multiple, intertwined factors\.

To this end, we conduct a series of comparative experiments to disentangle these factors, discovering four factors that reduce the prompt sensitivity of LLMs:\(1\)Supervised fine\-tuning reduces the prompt sensitivity\. Instruct/chat models \(with supervised fine\-tuning\) exhibit lower prompt sensitivity than base models\.\(2\)LLMs with larger parameter numbers exhibit lower prompt sensitivity\.\(3\)Dense models are generally less sensitive than mixture\-of\-experts \(MoE\) models\.\(4\)Few\-shot learning considerably reduces prompt sensitivity compared to 0\-shot learning\.

More crucially, we explore and uncover a common underlying mechanism that explains how the four aforementioned factors reduce prompt sensitivity: they primarily reduce the instability of low\-order interactions \(i\.e\., interactions involving a small number of input variables\)\. This finding is counterintuitive, as our experiments demonstrate that high\-order interactions \(i\.e\., interactions involving many input variables\) tend to exhibit the highest sensitivity, while low\-order interactions are inherently less sensitive\. Unexpectedly, these factors further stabilize the already stable low\-order interactions, yet remain ineffective in addressing the more pronounced sensitivity of high\-order interactions\.

## 2Related Work

Prompt sensitivity of LLMs\.Previous studies\([45](https://arxiv.org/html/2608.18539#bib.bib13);[56](https://arxiv.org/html/2608.18539#bib.bib12);[41](https://arxiv.org/html/2608.18539#bib.bib5);[1](https://arxiv.org/html/2608.18539#bib.bib6)\)demonstrated that LLMs are highly sensitive to minor perturbations or semantically unimportant alterations to prompts, which can lead to significant performance variation\. Such prompt sensitivity presents a considerable risk to the reliability of LLMs\. Existing metrics\([41](https://arxiv.org/html/2608.18539#bib.bib5);[10](https://arxiv.org/html/2608.18539#bib.bib8);[29](https://arxiv.org/html/2608.18539#bib.bib9);[58](https://arxiv.org/html/2608.18539#bib.bib11);[8](https://arxiv.org/html/2608.18539#bib.bib14);[18](https://arxiv.org/html/2608.18539#bib.bib7);[35](https://arxiv.org/html/2608.18539#bib.bib10)\)for evaluating the prompt sensitivity of LLMs typically measure shifts in final outputs, such as task accuracy or output consistency\. However, these metrics are coarse\-grained and fail to probe the LLM’s internal logic\. In this paper, we propose a fine\-grained metric that evaluates prompt sensitivity based on interactions\. This framework enables us to uncover the underlying mechanisms of prompt sensitivity in LLMs\.

Using game\-theoretic interactions to explain DNNs\.Traditional methods for explanations\([47](https://arxiv.org/html/2608.18539#bib.bib15);[53](https://arxiv.org/html/2608.18539#bib.bib16)\)often lack mathematical guarantees of faithfulness, meaning their outputs may not accurately represent the internal logic employed by the DNN\. To this end,[36](https://arxiv.org/html/2608.18539#bib.bib18)proposed to use interactions between input variables to explain DNNs and provided a series of theoretical guarantees for the method’s validity\. Furthermore, it has been empirically discovered\([27](https://arxiv.org/html/2608.18539#bib.bib17)\)and theoretically proven\([39](https://arxiv.org/html/2608.18539#bib.bib19)\)that a DNN typically encodes only a sparse set of interactions\. At the application level, the interaction framework has proven effective in a wide range of complex tasks, including adversarial transferability\([17](https://arxiv.org/html/2608.18539#bib.bib21)\), model generalization\([11](https://arxiv.org/html/2608.18539#bib.bib20);[54](https://arxiv.org/html/2608.18539#bib.bib23)\), model training process\([37](https://arxiv.org/html/2608.18539#bib.bib25);[40](https://arxiv.org/html/2608.18539#bib.bib26)\), overfitting\([38](https://arxiv.org/html/2608.18539#bib.bib22)\)and other tasks\([42](https://arxiv.org/html/2608.18539#bib.bib56);[28](https://arxiv.org/html/2608.18539#bib.bib54);[50](https://arxiv.org/html/2608.18539#bib.bib55)\)\. In this paper, we use the interaction framework to analyze the prompt sensitivity of LLMs\.

## 3Interaction\-Based Analysis of the Prompt Sensitivity of LLMs

### 3\.1Preliminaries: Interactions

This subsection introduces the definition of interactions, as well as the mathematical guarantees of interaction\-based explanation\. Given a DNNvvand an input sentence𝒙\\bm\{x\}withnninput variables \(e\.g\., words\) indexed byN=\{1,2,…,n\}N=\\\{1,2,\\ldots,n\\\}, letv⁡\(𝒙\)∈ℝv\(\\bm\{x\}\)\\in\\mathbb\{R\}denote the scalar output of the DNN\. Here we setv⁡\(𝒙\)=log​p⁡\(y=y\*\|𝒙\)1−p⁡\(y=y\*\|𝒙\)∈ℝv\(\\bm\{x\}\)=\\text\{log\}\\frac\{p\(y=y^\{\\text\{\*\}\}\|\\bm\{x\}\)\}\{1\-p\(y=y^\{\\text\{\*\}\}\|\\bm\{x\}\)\}\\in\\mathbb\{R\}, wherep⁡\(y=y\*\|𝒙\)p\(y=y^\{\\text\{\*\}\}\|\\bm\{x\}\)represents the probability of generating the ground truth tokeny\*y^\{\\text\{\*\}\}given the input𝒙\\bm\{x\}\. We define a surrogate logical modelϕ⁡\(𝒙\)\\phi\(\\bm\{x\}\)to match the scalar outputv⁡\(𝒙\)v\(\\bm\{x\}\)of the DNN\. Recent studies\([27](https://arxiv.org/html/2608.18539#bib.bib17);[36](https://arxiv.org/html/2608.18539#bib.bib18)\)have proven Theorem[3\.1](https://arxiv.org/html/2608.18539#S3.Thmtheorem1), which shows that the output scorev⁡\(𝒙\)v\(\\bm\{x\}\)on any randomly masked111It is common to use a specific token or embedding to mask input variables of a DNN,e\.g\., replacing the target token with a specific\[MASK\]token\. Please see Appendix[A](https://arxiv.org/html/2608.18539#A1)for details\.input𝒙T\\bm\{x\}\_\{T\}can be accurately calculated by the following surrogate logical modelϕ⁡\(⋅\)\\phi\(\\cdot\)\.

ϕ⁡\(𝒙T\)≜ϕ⁡\(𝒙∅\)\+∑S⊆N𝟙​\(S∣𝒙T\)⋅IS,\\small\\phi\(\\bm\{x\}\_\{T\}\)\\triangleq\\phi\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq N\}\\mathds\{1\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)\\cdot I\_\{S\},\(1\)where the AND trigger function𝟙​\(S∣𝒙T\)∈\{0,1\}\\mathds\{1\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)\\in\\\{0,1\\\}represents anAND relationshipbetween input variables inSS, which can also be termedAND interaction pattern\. The scalar weightISI\_\{S\}quantifies the effect of an AND relationship, which can also be termedinteraction effect\. An AND relationship is activated only by the joint presence of all input variables in the setSS,i\.e\., all input variables inSSare not masked\. For instance, given the input sentence𝒙=“He is a green hand,”\\bm\{x\}=\\textit\{\`\`He\\ is\\ a\\ green\\ hand,''\}the co\-occurrence of the input variables in the setS=\{green,hand\}S=\\\{\\textit\{green\},\\textit\{hand\}\\\}contributes a numerical effectISI\_\{S\}that pushes the surrogate logical model’s inference towards the semantic meaning of“beginner\.”If an AND interactionSSis triggered,i\.e\.,𝟙​\(S∣𝒙T\)=1\\mathds\{1\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)=1, the corresponding interaction effectISI\_\{S\}is added to the output of the logical model\. Otherwise, if any word inSSis masked and the AND interaction is not triggered,i\.e\.,𝟙​\(S∣𝒙T\)=0\\mathds\{1\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)=0, its corresponding interaction effectISI\_\{S\}is not added to the output of the logical model\.𝒙∅\\bm\{x\}\_\{\\emptyset\}represents that all input variables inNNare masked\.

###### Theorem 3\.1\(Universal matching property, proven in Appendix[B](https://arxiv.org/html/2608.18539#A2)\)\.

Given an input𝐱\\bm\{x\}withnninput variables, we randomly mask any combinations of input variables to generate2n2^\{n\}masked inputs\{𝐱T∣T⊆N\}\\\{\\bm\{x\}\_\{T\}\\mid T\\subseteq N\\\}\. For every masked input𝐱T\\bm\{x\}\_\{T\}, when the scalar weightISI\_\{S\}in the logical modelϕ⁡\(⋅\)\\phi\(\\cdot\)are set toIS=∑S′⊆S\(−1\)\|S\|−\|S′\|⋅v⁡\(𝐱S′\),ϕ⁡\(𝐱∅\)=v⁡\(𝐱∅\)I\_\{S\}=\\sum\\nolimits\_\{S^\{\\prime\}\\subseteq S\}\(\-1\)^\{\|S\|\-\|S^\{\\prime\}\|\}\\cdot v\(\\bm\{x\}\_\{S^\{\\prime\}\}\),\\phi\(\\bm\{x\}\_\{\\emptyset\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\), the output of the logical modelϕ⁡\(⋅\)\\phi\(\\cdot\)can always match the DNN’s output scorev⁡\(⋅\)v\(\\cdot\)\.

∀T⊆N,v⁡\(𝒙T\)=ϕ⁡\(𝒙T\)\\small\\forall\\ T\\subseteq N,\\ \\ v\(\\bm\{x\}\_\{T\}\)=\\phi\(\\bm\{x\}\_\{T\}\)\(2\)

Figure 2:\(a\) Verifying the sparsity of interactions\. We show absolute values of normalized interactions in a descending order\. LLMs all encode a small number of salient interactions, while most of the interaction effects are negligible\. \(b\) Verifying the quality of universal matching for any2n2^\{n\}masked inputs\. The red line plots outputs of the LLM in an ascending order\.In addition,the sparsity property also provides a theoretical guarantee for the faithfulness of interaction\-based explanation\. The sparsity property shows that a DNN only encodes a sparse set of interactions with salient effects\. That is, only a small subset of all2n2^\{n\}interactions in Theorem[3\.1](https://arxiv.org/html/2608.18539#S3.Thmtheorem1), termed salient interactions, have a significant impact on the logical model’s output\. In contrast, the majority of interactions have negligible effects and are considered noise patterns\. The sparsity property of AND interactions has been proven by[39](https://arxiv.org/html/2608.18539#bib.bib19)\. We follow[27](https://arxiv.org/html/2608.18539#bib.bib17)to extract AND\-OR interactions222The OR interaction is proved to be a specific AND interaction\. Please see Appendix[C](https://arxiv.org/html/2608.18539#A3)for proof and Appendix[D](https://arxiv.org/html/2608.18539#A4)for how to extract OR interactions\.from input variables\. Such a technique has proven effective in pursuing higher sparsity of interactions, supported by both theoretical proofs\([27](https://arxiv.org/html/2608.18539#bib.bib17)\)and extensive empirical validation\([36](https://arxiv.org/html/2608.18539#bib.bib18);[54](https://arxiv.org/html/2608.18539#bib.bib23)\)\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/case_study_detail.png)Figure 3:\(a\) Case studies of interaction\-level analysis\. The plots compare the interaction effects encoded by the LLM for the same input𝒙\\bm\{x\}under two semantically equivalent prompt templates,TT\(blue\) andT^\\hat\{T\}\(red\)\. The bottom row shows examples where the LLM’s output remains the same; the top row shows examples where it changes\. Results show that interaction effects are highly unstable, even when the output of the LLM remains the same\. \(b\) The distribution of salient interactions types of various LLMs\.
### 3\.2Verifying the Faithfulness of Considering Interactions as Inference Patterns Used by LLMs

Before utilizing the interaction framework to analyze the prompt sensitivity of LLMs, we need totheoretically prove and experimentally validatethe faithfulness of using interactions to explain LLMs\.In theory, Theorem[3\.1](https://arxiv.org/html/2608.18539#S3.Thmtheorem1)guarantees that the surrogate logical model’s outputϕ⁡\(⋅\)\\phi\(\\cdot\)can always match the LLM’s outputv⁡\(⋅\)v\(\\cdot\)for all2n2^\{n\}masked samples\. Since the logical model’s outputϕ⁡\(⋅\)\\phi\(\\cdot\)is composed entirely of the sum of all interaction effects as defined in Eq\. \([1](https://arxiv.org/html/2608.18539#S3.E1)\),we can consider interactions as the detailed inference patterns that constitute the LLM’s internal logic\. Therefore, we can evaluate the prompt sensitivity of LLMs by measuring the instability of interaction patterns\.

In practice, we conduct experiments to verify whether LLMs encode sparse interactions\. Consider the multiple choice question \(MCQ\) in Figure[1](https://arxiv.org/html/2608.18539#S1.F1)as an example\. LetQQdenote the set of all words in the question,e\.g\.,QQ= \{Which, one, of, these, animals, does, NOT, lay, eggs\},MMdenote the set of all words in the options,e\.g\.,MM= \{chickens, dogs, frogs, turtles\}, andTTdenote the set of all words in the prompt template,e\.g\.,TT= \{Answers:, A\., B\., C\., D\., Answer:\}\. Specifically, we use words333We use words instead of tokens as input variables because different LLMs may divide the same word into different tokens\. For example, Llama\-2\-7B tokenizes the word “Elements” into two tokens “Element” and “s”, while Qwen3\-8B treats it as one token\.inQ∪MQ\\cup Mas input variables and compute the interaction effectISI\_\{S\}of all interactionsS⊆Q∪MS\\subseteq Q\\cup M\. Meanwhile, we treat words in prompt templateTTas background context\. We follow[27](https://arxiv.org/html/2608.18539#bib.bib17)to extract AND and OR interactions\. Thus, given an input𝒙=Q∪M\\bm\{x\}=Q\\cup Mwithnnwords, we can obtain2n\+12^\{n\+1\}interactions, including2n2^\{n\}AND interactions and2n2^\{n\}OR interactions\. For each interaction effectISI\_\{S\}, we apply min\-max normalization to it\. Specifically,I~S≜sgn​\(IS\)⋅\|IS\|−MinMax−Min\\widetilde\{I\}\_\{S\}\\triangleq\\textit\{sgn\}\(I\_\{S\}\)\\cdot\\frac\{\|I\_\{S\}\|\-\\textit\{Min\}\}\{\\textit\{Max\}\-\\textit\{Min\}\}, whereMinandMaxare the minimum and maximum absolute values of all2n\+12^\{n\+1\}interaction effects;sgn​\(IS\)=IS\|IS\|\\textit\{sgn\}\(I\_\{S\}\)=\\frac\{I\_\{S\}\}\{\|I\_\{S\}\|\}represents the sign ofISI\_\{S\}\. Figure[2](https://arxiv.org/html/2608.18539#S3.F2)\(a\) shows the distribution of\|I~S\|\|\\widetilde\{I\}\_\{S\}\|\. Results444Results on more LLMs in Appendix[F](https://arxiv.org/html/2608.18539#A6)support the conclusion\.verify that only a small set of interactions have salient effects, while most of the interactions have negligible effects and can be considered as noise patterns\. Figure[2](https://arxiv.org/html/2608.18539#S3.F2)\(b\) compares the LLM’s true outputv⁡\(𝒙T\)v\(\\bm\{x\}\_\{T\}\)for all2n2^\{n\}masked inputs against the logical model using only the most salient interactions\. Even when using the top 3% or top 5% of all interactions, the matching error is minimal\. This empirically demonstrates that the LLM’s output can be faithfully approximated by a small set of salient interactions\.

### 3\.3Using Interactions as a Fine\-Grained Tool to Analyze the Prompt Sensitivity of LLMs

Based on the above verification, we deploy interactions as a fine\-grained analytical tool\. Specifically, for an LLM, we visualize the changes in salient interactions for the same input under a pair of prompt templates\. As Figure[3](https://arxiv.org/html/2608.18539#S3.F3)\(a\) shows, when the LLM’s outputs are different, nearly half of the salient interactions reverse their sign \(from positive to negative, or vice versa\), another 30%\-40% change significantly in magnitude, and only a small fraction \(10%\-20%\) remain stable\. Strikingly, even when the LLM’s output remains the same, the majority \(60%\-80%\) of the salient interactions are still unstable\. This offers preliminary evidence that semantically irrelevant alterations to the prompt template can lead to significant changes in the salient interaction patterns\.

To systematically analyze this instability beyond individual cases, we classify interactions into three distinct types: 1\)Opposite sign: The interaction effect reverses its sign \(e\.g\., from positive to negative\)\. The sign change represents the most severe form of instability, as it can completely reverse the LLM’s internal logic\. 2\)Same sign & different effect: The interaction maintains its positive or negative influence, but its magnitude changes substantially \(i\.e\., its effect is more than doubled or less than halved\)\. 3\)Same sign & similar effect: The interaction’s sign and magnitude both remain stable, which represents robust, stable interactions\. Figure[3](https://arxiv.org/html/2608.18539#S3.F3)\(b\) plots the distribution of three interaction types over all samples, conditioned on whether the LLM’s final output changes or remains the same across a pair of prompts\. When the LLM generates different outputs, the interaction patterns are highly unstable\.Opposite signinteractions account for approximately 50%, meaning nearly half of salient interactions reverse the sign of their effect\. The truly stableSame sign & similar effectinteractions account for a mere 2\.7%\-6\.8%, while the remaining 42\.6%\-47\.4% of interactions, though maintaining their sign, change significantly in magnitude\. More alarmingly, even when the final output of the LLM remains the same, the instability of interactions still exists\. While the situation improves, the majority of interactions still fall into the two unstable categories\.\.

The results demonstrate that output\-level analysis is insufficient to capture the unreliable internal patterns of LLMs\. Conversely, our interaction\-based analysis offers a fine\-grained lens to uncover latent instability of LLMs, offering a new analytical tool for quantifying the ratio of stable and unstable interactions on a per\-sample basis for any LLM\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/all_models.png)Figure 4:Prompt sensitivity of all the 50 open\-source LLMs in an ascending order\.

## 4Evaluating and Analyzing the Prompt Sensitivity of LLMs

### 4\.1Evaluating Interaction\-Based Prompt Sensitivity

We propose an interaction\-based metric to measure the prompt sensitivity of LLMs\. Given an MCQ dataset𝒟\\mathcal\{D\}, for any input𝒙∈𝒟\\bm\{x\}\\in\\mathcal\{D\}, it is composed of the questionQQand the optionsMM,i\.e\.,𝒙=Q∪M\\bm\{x\}=Q\\cup M\. Given a prompt templateTT, we make minor changes toTTand obtain a modified prompt templateT^\\hat\{T\}\. By applying the pair of prompt templatesTTandT^\\hat\{T\}to the same inputQ∪MQ\\cup M, we can construct two similar prompts\. The LLM is supposed to extract similar salient interactions fromQ∪MQ\\cup Mwhen processing these two prompts becauseTTandT^\\hat\{T\}have the same semantic meaning\. Thus, letΩsalient​\(𝒙\|T\)=\{S∈Ω⁡\(𝒙\)∣\|I~S​\(𝒙\|T\)\|\>τ\}\\Omega\_\{\\text\{salient\}\}\(\\bm\{x\}\|T\)=\\\{S\\in\\Omega\(\\bm\{x\}\)\\mid\|\\widetilde\{I\}\_\{S\}\(\\bm\{x\}\|T\)\|\>\\tau\\\}represent the set ofsalient interactionsextracted from𝒙\\bm\{x\}given the prompt templateTT\. Hereτ\\tauis a threshold used to distinguish salient interactions from noise patterns\. In the main paper, we setτ\\tauas 0\.1 for all experiments\. Please see Appendix[F\.8](https://arxiv.org/html/2608.18539#A6.SS8)for hyperparameter experiments ofτ\\tau\. Similarly, letΩsalient​\(𝒙\|T^\)=\{S∈Ω⁡\(𝒙\)∣\|I~S​\(𝒙\|T^\)\|\>τ\}\\Omega\_\{\\text\{salient\}\}\(\\bm\{x\}\|\\hat\{T\}\)=\\\{S\\in\\Omega\(\\bm\{x\}\)\\mid\|\\widetilde\{I\}\_\{S\}\(\\bm\{x\}\|\\hat\{T\}\)\|\>\\tau\\\}represent the set ofsalient interactionsextracted from𝒙\\bm\{x\}given the prompt templateT^\\hat\{T\}\. Therefore, on the dataset𝒟\\mathcal\{D\}, we define the LLM’sInteraction\-based Prompt SensitivityasIPS\.

IPS≜𝔼𝒙​\[𝔼T,T^​\[1\|Ωunion\|​∑S∈Ωunion\|ℐ~S​\(𝒙\|T\)−ℐ~S​\(𝒙\|T^\)\|\|I~S​\(𝒙\|T\)\|\+\|I~S​\(𝒙\|T^\)\|2\]\],\\small\\textit\{IPS\}\\triangleq\\mathbb\{E\}\_\{\\bm\{x\}\}\\left\[\\mathbb\{E\}\_\{T,\\hat\{T\}\}\\left\[\\frac\{1\}\{\|\{\\Omega\}\_\{\\text\{union\}\}\|\}\\sum\_\{S\\in\{\\Omega\}\_\{\\text\{union\}\}\}\\frac\{\|\\mathcal\{\\widetilde\{I\}\}\_\{S\}\(\\bm\{x\}\|T\)\-\\mathcal\{\\widetilde\{I\}\}\_\{S\}\(\\bm\{x\}\|\\hat\{T\}\)\|\}\{\\frac\{\|\\widetilde\{I\}\_\{S\}\(\\bm\{x\}\|T\)\|\+\|\\widetilde\{I\}\_\{S\}\(\\bm\{x\}\|\\hat\{T\}\)\|\}\{2\}\}\\right\]\\right\],\(3\)whereΩunion=Ωsalient​\(𝒙\|T\)∪Ωsalient​\(𝒙\|T^\)\{\\Omega\}\_\{\\text\{union\}\}=\\Omega\_\{\\text\{salient\}\}\(\\bm\{x\}\|T\)\\cup\\Omega\_\{\\text\{salient\}\}\(\\bm\{x\}\|\\hat\{T\}\)is a unified set by taking the union of the two salient setsΩsalient​\(𝒙\|T\)\\Omega\_\{\\text\{salient\}\}\(\\bm\{x\}\|T\)andΩsalient​\(𝒙\|T^\)\\Omega\_\{\\text\{salient\}\}\(\\bm\{x\}\|\\hat\{T\}\); the outer expectation,𝔼𝒙\\mathbb\{E\}\_\{\\bm\{x\}\}, represents an averaging over all inputs𝒙∈𝒟\\bm\{x\}\\in\\mathcal\{D\}, and the inner expectation,𝔼T,T^\\mathbb\{E\}\_\{T,\\hat\{T\}\}, represents an averaging over all pairs of prompt templates\(T,T^\)\(T,\\hat\{T\}\)\. This metric evaluates prompt sensitivity of LLMs by calculating the symmetric mean absolute percentage error of salient interactions over all samples in the dataset𝒟\\mathcal\{D\}\.

Models and Datasets\.We conduct experiments on50 open\-source LLMs from 6 model families\. This diverse set includes 10 LLMs from the Llama family: Llama\-2, Llama\-3, Llama\-MoE\([48](https://arxiv.org/html/2608.18539#bib.bib29);[21](https://arxiv.org/html/2608.18539#bib.bib30);[57](https://arxiv.org/html/2608.18539#bib.bib31);[34](https://arxiv.org/html/2608.18539#bib.bib32)\); 4 LLMs from the Mistral family: Mistral, Mixtral\([25](https://arxiv.org/html/2608.18539#bib.bib33);[24](https://arxiv.org/html/2608.18539#bib.bib34)\); 25 LLMs from the Qwen family: Qwen2, Qwen2\.5, Qwen3, Qwen1\.5\-MoE, Qwen3\-30B\-A3B\([52](https://arxiv.org/html/2608.18539#bib.bib35);[51](https://arxiv.org/html/2608.18539#bib.bib36)\); 9 LLMs from the OLMo family: OLMo, OLMo\-2, OLMoE\([22](https://arxiv.org/html/2608.18539#bib.bib37);[32](https://arxiv.org/html/2608.18539#bib.bib39);[31](https://arxiv.org/html/2608.18539#bib.bib38)\); 2 LLMs from the InternLM family: InternLM2\([7](https://arxiv.org/html/2608.18539#bib.bib40)\)\. We evaluate all LLMs ontwo widely used MCQ benchmarks: ARC\([13](https://arxiv.org/html/2608.18539#bib.bib27)\)and MMLU\([23](https://arxiv.org/html/2608.18539#bib.bib28)\)\. Experiments onopen\-ended tasksare shown in Section[4\.3](https://arxiv.org/html/2608.18539#S4.SS3)\. For each input, we apply five distinct prompt templates to it\. These templates maintain the same core content and differ only in minor formatting details, such as letter case and separators\. For masking words in the input sentences, we follow the approach of[12](https://arxiv.org/html/2608.18539#bib.bib24)and utilize a certain\[MASK\]token for each LLM\. A comprehensive list of all LLMs, along with the specific prompt templates, mask tokens used in experiments is provided in Appendix[E](https://arxiv.org/html/2608.18539#A5)\.

We apply the IPS metric to evaluate the prompt sensitivity of 50 open\-source LLMs\. As shown in Figure[4](https://arxiv.org/html/2608.18539#S3.F4)\(a\), we observe a wide variance in IPS, ranging from 1\.268 \(Qwen2\.5\-72B\-Instruct\) to 1\.752 \(Mistral\-7B\-v0\.3\)\. However, no LLM series or families achieve a complete victory\. The distribution indicates that an LLM’s prompt sensitivity is influenced by multiple underlying factors\. Identifying these factors is of great significance for the robustness research of LLMs\.

To validate the reliability of the IPS metric, we investigate its correlation with the traditional metrics output consistency, which is defined as the proportion of samples where the LLM generates identical predictions across different prompt templates\. Figure[4](https://arxiv.org/html/2608.18539#S3.F4)\(b\) shows that the IPS score exhibits a negative correlation with output consistency\. This result confirms thatIPS aligns well with the output consistency while offering a more fine\-grained perspective to quantify prompt sensitivity beyond output matching\.

Robustness to the thresholdτ\\tau\.In the main paper, we set the thresholdτ\\tauas 0\.1 for all experiments\. To verify that our findings are robust to the selection ofτ\\tau, we compute interactions across 16 distinct threshold values, ranging from 0\.05 to 0\.20 with a step size of 0\.01\. We evaluate the consistency of IPS rankings across these thresholds, observing an average Spearman’s rank correlation of0\.9905and an average Pearson correlation of0\.9957\(Appendix[F\.8\.1](https://arxiv.org/html/2608.18539#A6.SS8.SSS1)\)\. These near\-perfect correlations indicate that therelativeranking among LLMs remains highly stable\. More crucially, we conduct all the experiment withτ=0\.05\\tau=0\.05andτ=0\.15\\tau=0\.15\. As detailed in Appendix[F\.8\.2](https://arxiv.org/html/2608.18539#A6.SS8.SSS2), the main conclusions remain same asτ=0\.1\\tau=0\.1, proving that our findings are robust to differentτ\\tau\.

Figure 5:A comparison of the prompt sensitivity between instruct/chat models and base models\. Results show that instruct/chat models are less sensitive than corresponding base models\.
### 4\.2Analyzing the Factors Impacting Prompt Sensitivity

In this section, we investigate four factors that might influence the prompt sensitivity of LLMs, including \(1\) supervised fine\-tuning, \(2\) model scales, \(3\) model architectures, and \(4\) prompting methods\.

Factor 1: instruct/chat models vs\. base models\.Supervised fine\-tuning \(e\.g\., instruction tuning\) is now a common practice to align base models with human preferences for certain tasks, yielding models often referred to as instruct or chat models \(for different tasks or purposes\)\. Therefore, we investigate the impact of supervised fine\-tuning by comparing the prompt sensitivity of instruct/chat models with base models\. Results555Results on the MMLU dataset in Appendix[F](https://arxiv.org/html/2608.18539#A6)exhibit the same conclusion\.on the ARC dataset in Figure[5](https://arxiv.org/html/2608.18539#S4.F5)show that almost all instruct/chat models exhibit lower prompt sensitivity than their corresponding base models\. This demonstrates that supervised fine\-tuning enables the LLM to encode more stable interactions\. A valid explanation is that base models are pre\-trained on unstructured and raw texts, but instruct/chat models are further fine\-tuned on instruction\-response datasets or dialogue datasets\. Thus, instruct/chat models can precisely understand the function of prompt templates and focus on the task\-relevant inputs\.

Figure 6:A comparison of prompt sensitivity across different model scales\. As the model scale increases, the prompt sensitivity within a model series systematically decreases\.Factor 2: model scales\.We investigate the relationship between the model scale \(i\.e\., the number of parameters\) and the prompt sensitivity\. Results[5](https://arxiv.org/html/2608.18539#footnote5)on the ARC dataset in Figure[6](https://arxiv.org/html/2608.18539#S4.F6)show that within the same model series, as the model scale increases, the overall prompt sensitivity systematically decreases\. This demonstrates that larger LLMs encode more stable interactions, making them less susceptible to superficial changes in the prompt template\.

Factor 3: dense models vs\. MoE models\.MoE models scale up model capacity with minimal computational cost by dynamically activating different subsets of“expert”sub\-networks\([6](https://arxiv.org/html/2608.18539#bib.bib41)\)\. In contrast, all parameters in dense models participate in every computation\. We aim to investigate the impact of model architectures by comparing the prompt sensitivity of dense models with MoE models\. Results[5](https://arxiv.org/html/2608.18539#footnote5)on the ARC dataset in Figure[7](https://arxiv.org/html/2608.18539#S4.F7)show that in model families including Llama\-2, Llama\-3, Qwen and Olmo, all MoE models exhibit higher prompt sensitivity than dense models\. This suggests that MoE models encode more unstable interactions\. A potential confounding factor is the number of active parameters\. Given that smaller models are more sensitive \(Factor 2\), one might attribute MoE instability to lower active parameters\. To mitigate the influence of active parameters, we conduct controlled variable analysis\. Results \(Table[7](https://arxiv.org/html/2608.18539#A9.T7), Appendix[I](https://arxiv.org/html/2608.18539#A9)\) confirms thatMoE models remain more sensitive than dense models even with similar active parameters\.We attribute this to the dynamic routing mechanism\. For different prompt templates, the gating network may route the input to different experts so that it is processed by different sub\-networks, leading to different interactions encoded and thus higher sensitivity\. In conclusion, while MoE models achieve impressive performance with reduced computational overhead, this benefit comes at the cost of weaker stability\.

Figure 7:A Comparison of prompt sensitivity between MoE models and dense models\. Generally, MoE models tend to be more sensitive than dense models in the same model family\.Figure 8:A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\. The drop in prompt sensitivity is substantial from 0\-shot to 1\-shot\.Factor 4: few\-shot learning vs\. 0\-shot learning\.Few\-shot learning is utilized to improve LLMs’ performance by providing in\-context examples to better specify the task\([49](https://arxiv.org/html/2608.18539#bib.bib42)\)\. We investigate the impact of prompting methods on prompt sensitivity by comparing 0\-shot learning with few\-shot learning666See Appendix[E](https://arxiv.org/html/2608.18539#A5)for the prompt templates and settings of few\-shot learning\.\. Results[5](https://arxiv.org/html/2608.18539#footnote5)on the ARC dataset in Figure[8](https://arxiv.org/html/2608.18539#S4.F8)show that incorporating in\-context examples leads to a significant reduction in prompt sensitivity across all tested LLMs\. For most of the LLMs, the most substantial drop occurs when moving from 0\-shot to 1\-shot, while adding more in\-context examples yields slower reductions\. This suggests that even a single example is sufficient to establish the LLM’s understanding of the task, leading it to ignore superficial template variations and focus on the core input\.

Figure 9:Prompt sensitivity of LLMs for different order types\.![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_0.1_new.png)Figure 10:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 11:A comparison of prompt sensitivity of different order types across different model scales\.
### 4\.3Explore the Underlying Mechanisms of Improved Stability for All Factors

In this section, we aim to explorewhether there exists a common reason to explain the underlying mechanisms by which the four aforementioned factors reduce the prompt sensitivity of LLMs\. Specifically, we analyze the prompt sensitivity of different types of interactions, so as to reveal the source of the LLM’s prompt sensitivity\. To this end, we analyze the sensitivity of interactions with different complexities, which are defined as the orders of interactions\. The order of an interactionSSis defined as the number of input variables involved,i\.e\.,\|S\|\|S\|\. An interaction with high order indicates an intricate relationship including many input variables, while an interaction with low order represents a simple relationship including few input variables\. We further define three types of prompt sensitivity metrics corresponding to different types of interaction orders\. Specifically, we partition interactions inΩunion\{\\Omega\}\_\{\\text\{union\}\}in Eq\. \([3](https://arxiv.org/html/2608.18539#S4.E3)\) into three distinct groups based on their orders: low\-order, mid\-order, and high\-order\. Given an input𝒙\\bm\{x\}withnnwords,Ωunionlow≜\{S∈Ωunion∣1≤\|S\|≤⌊13​n⌋\},Ωunionmid≜\{S∈Ωunion∣⌊13​n⌋<\|S\|≤⌊23​n⌋\},Ωunionhigh≜\{S∈Ωunion∣⌊23​n⌋<\|S\|≤n\}\{\\Omega\}^\{\\textit\{low\}\}\_\{\\text\{union\}\}\\triangleq\\\{S\\in\{\\Omega\}\_\{\\text\{union\}\}\\mid 1\\leq\|S\|\\leq\\lfloor\\frac\{1\}\{3\}n\\rfloor\\\},\{\\Omega\}^\{\\textit\{mid\}\}\_\{\\text\{union\}\}\\triangleq\\\{S\\in\{\\Omega\}\_\{\\text\{union\}\}\\mid\\lfloor\\frac\{1\}\{3\}n\\rfloor<\|S\|\\leq\\lfloor\\frac\{2\}\{3\}n\\rfloor\\\},\{\\Omega\}^\{\\textit\{high\}\}\_\{\\text\{union\}\}\\triangleq\\\{S\\in\{\\Omega\}\_\{\\text\{union\}\}\\mid\\lfloor\\frac\{2\}\{3\}n\\rfloor<\|S\|\\leq n\\\}\. Then we calculate prompt sensitivity for low\-order, mid\-order, and high\-order interactions, asIPSlow\\textit\{IPS\}^\{\\textit\{low\}\},IPSmid\\textit\{IPS\}^\{\\textit\{mid\}\},IPShigh\\textit\{IPS\}^\{\\textit\{high\}\}\.

Figure[9](https://arxiv.org/html/2608.18539#S4.F9)presents the prompt sensitivity of different order types on the ARC dataset\. Results[4](https://arxiv.org/html/2608.18539#footnote4)show that the prompt sensitivity of low\-order interactions is the lowest, followed by mid\-order, while high\-order interactions exhibit the highest prompt sensitivity\. This indicates that low\-order interactions encoded by LLMs are relatively stable when faced with subtle changes to prompt templates,i\.e\., simple interaction patterns are more robust\. Conversely, the high sensitivity of high\-order interactions reveals that the LLMs’ internal representation of complex patterns is highly unstable\.

Explaining why the four factors can reduce prompt sensitivity\.Inspired by the results in Figure[9](https://arxiv.org/html/2608.18539#S4.F9), we now investigate how the four aforementioned factors influence the prompt sensitivity of low\-, mid\-, and high\-order interactions\. This order\-level analysis aims to reveal the common mechanism by which these factors reduce the LLM’s prompt sensitivity\. For each factor, we quantify its effect on low\-, mid\-, and high\-order interactions by computing the relative change in IPS between the LLM with the factor and its counterpart without it\. Specifically, giventype∈\{low,mid,high\}\\textit\{type\}\\in\\\{\\textit\{low\},\\textit\{mid\},\\textit\{high\}\\\}, the relative change is defined asΔ​IPStype=\(IPSAtype−IPSBtype\)/IPSBtype\\Delta\\textit\{IPS\}^\{\\textit\{type\}\}=\(\\textit\{IPS\}^\{\\textit\{type\}\}\_\{\\text\{A\}\}\-\\textit\{IPS\}^\{\\textit\{type\}\}\_\{\\text\{B\}\}\)/\\textit\{IPS\}^\{\\textit\{type\}\}\_\{\\text\{B\}\}\. ForFactor 1\(fine\-tuned vs\. base\), A and B are fine\-tuned and base models, respectively\. ForFactor 3\(dense vs\. MoE\), A and B are dense and MoE models, with the finalΔ​IPStype\\Delta\\textit\{IPS\}^\{\\textit\{type\}\}being the average over all pairs of a specific dense model and a specific MoE model within a model family\. ForFactor 4\(few\-shot vs\. 0\-shot\), A and B are x\-shot \(x∈\{1,2,3\}x\\in\{\\\{1,2,3\\\}\}\) and 0\-shot learning\. The relative change metric is unsuitable forFactor 2\(model scales\), as scale is a continuous variable\. Instead, we directly analyze the trend of IPS values as model parameters increase\.

Results[4](https://arxiv.org/html/2608.18539#footnote4)shown in Figure[10](https://arxiv.org/html/2608.18539#S4.F10)and Figure[11](https://arxiv.org/html/2608.18539#S4.F11)converge on a common explanation for how the four factors reduce prompt sensitivity\.The most significant reduction in prompt sensitivity is consistently observed in low\-order interactions\.An obvious, though less pronounced, decrease is also seen at the mid\-order level\. In contrast, the change of the sensitivity of high\-order interactions is relatively minimal, remaining at a high level\. It indicates that the stability of low\-order interactions is critical to the overall robustness of LLMs\.

This phenomenon is unexpected\. Although results in Figure[9](https://arxiv.org/html/2608.18539#S4.F9)show that low\-order interactions are naturally more robust than other types of interactions, the four factors above still significantly reduce the sensitivity of low\-order interactions\. Instead, they fail to reduce the sensitivity of high\-order interactions, which are inherently the most sensitive\. This phenomenon indicates that stable low\-order interactions are much easier for LLMs to learn, while it is difficult for LLMs to make high\-order interactions more stable\.

Robustness on open\-ended tasks\.To verify the robustness of our findings, we conduct additional experiments on open\-ended generation tasks\. We utilize the Dolly\-15k dataset\([14](https://arxiv.org/html/2608.18539#bib.bib52)\), which contains a diverse range of non\-MCQ tasks, including open Q&A, classification, and others\. The results of this analysis, detailed in Appendix[K](https://arxiv.org/html/2608.18539#A11), consistently verify our main conclusions drawn from the MCQ experiments\. This strongly suggests that the four factors and the underlying mechanisms of prompt sensitivity we have uncovered can be generalized to open\-ended questions\.

Robustness to more complex prompt perturbations\.In this experimental setup, we use the Dolly\-15k dataset and introduce more complex prompt perturbations, specifically semantic paraphrases and instruction reordering \(detailed in Appendix[L\.1](https://arxiv.org/html/2608.18539#A12.SS1)\), to test the robustness of our conclusions\. The results in Figures[12](https://arxiv.org/html/2608.18539#S4.F12),[13](https://arxiv.org/html/2608.18539#S4.F13), and[14](https://arxiv.org/html/2608.18539#S4.F14)consistently affirm our conclusions drawn from the template\-based experiments\. This indicates that our conclusions are robust to more complex prompt perturbations\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/open-ended-factor.png)Figure 12:\(a\) A comparison of the prompt sensitivity between instruct/chat models and base models\. \(b\) A comparison of prompt sensitivity across different model scales\. \(c\) A comparison of prompt sensitivity between MoE models and dense models\. \(d\) A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\.![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_dolly2.png)Figure 13:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 14:A comparison of prompt sensitivity at the order\-level across different model scales\.Strategies for reducing the computational cost\.To reduce the computational cost of the interaction framework, recent studies\([11](https://arxiv.org/html/2608.18539#bib.bib20);[12](https://arxiv.org/html/2608.18539#bib.bib24)\)used the following two strategies:\(1\) Select informative words as input variables while treating uninformative ones \(e\.g\., stop words\) as fixed background context\.\(2\) Merge related words into combined phrases as input variables\. The specific selection strategies are detailed in Appendix[K\.2](https://arxiv.org/html/2608.18539#A11.SS2)\. Results in Appendix[K\.3](https://arxiv.org/html/2608.18539#A11.SS3)show that using the above two strategies on open\-ended tasks yields the same conclusions\. In addition to selection strategies, techniques specifically designed for efficiently computing sparse interactions to bypass exhaustiveO⁡\(2n\)O\(2^\{n\}\)evaluations\([26](https://arxiv.org/html/2608.18539#bib.bib57);[5](https://arxiv.org/html/2608.18539#bib.bib58)\)represent a clear path for reducing computational cost\. Further details are provided in Appendix[J](https://arxiv.org/html/2608.18539#A10)\.

## 5Conclusion

In this paper, we propose an interaction\-based metric to evaluate the prompt sensitivity of LLMs\. We discover that employing supervised fine\-tuning, increasing model scale, using dense over MoE architectures, and applying few\-shot learning all serve to reduce the prompt sensitivity of LLMs\. Our findings offer novel insights into both model designs and prompting methods for improving the robustness of LLMs\. More crucially, we find that these factors achieve lower sensitivity primarily by reducing the sensitivity of low\-order interactions, while the prompt sensitivity of high\-order interactions remains at a relatively high level\. In future studies, new training methods could be designed to increase the LLM’s reliance on stable low\-order interactions or, alternatively, to reduce the instability of high\-order interactions\.

## Acknowledgements

This work is partially supported by the Shanghai Science and Technology Commission \(No\. 25511102900\), the National Nature Science Foundation of China \(No\.62376199,62576249\), and the Shanghai Municipal Education Commission \(No\. 24CGA20\)\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- Alzahraniet al\.\(2024\)N\. Alzahrani, H\. Alyahya, Y\. Alnumay, S\. AlRashed, S\. Alsubaie, Y\. Almushayqih, F\. Mirza, N\. Alotaibi, N\. Al\-Twairesh, A\. Alowisheq, M\. S\. Bari, and H\. KhanWhen benchmarks are targets: revealing the sensitivity of large language model leaderboards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13787–13805\.External Links:[Link](https://aclanthology.org/2024.acl-long.744/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.744)Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1),[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Anconaet al\.\(2019\)M\. Ancona, C\. Oztireli, and M\. GrossExplaining deep neural networks with a polynomial time algorithm for Shapley value approximation\.InInternational conference on machine learning,pp\. 272–281\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Annepaka and Pakray \(2024\)Y\. Annepaka and P\. PakrayLarge language models: a survey of their development, capabilities, and applications\.Knowledge and Information Systems,pp\. 1–56\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1)\.
- Bird \(2006\)S\. BirdNLTK: the natural language toolkit\.InProceedings of the COLING/ACL 2006 interactive presentation sessions,pp\. 69–72\.Cited by:[§K\.2](https://arxiv.org/html/2608.18539#A11.SS2.p2.1)\.
- Butleret al\.\(2026\)L\. Butler, A\. Agarwal, J\. S\. Kang, Y\. E\. Erginbas, B\. Yu, and K\. RamchandranProxySPEX: inference\-efficient interpretability via sparse feature interactions in LLMs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=KI8qan2EA7)Cited by:[Appendix J](https://arxiv.org/html/2608.18539#A10.p6.1),[§4\.3](https://arxiv.org/html/2608.18539#S4.SS3.p8.1)\.
- Caiet al\.\(2025\)W\. Cai, J\. Jiang, F\. Wang, J\. Tang, S\. Kim, and J\. HuangA survey on mixture of experts in large language models\.IEEE Transactions on Knowledge and Data Engineering\.Cited by:[§4\.2](https://arxiv.org/html/2608.18539#S4.SS2.p4.1)\.
- Caiet al\.\(2024\)Z\. Cai, M\. Cao, H\. Chen, K\. Chen, K\. Chen, X\. Chen, X\. Chen, Z\. Chen, Z\. Chen, P\. Chu,et al\.InternLM2 technical report\.arXiv preprint arXiv:2403\.17297\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Caoet al\.\(2024\)B\. Cao, D\. Cai, Z\. Zhang, Y\. Zou, and W\. LamOn the worst prompt performance of large language models\.Advances in Neural Information Processing Systems37,pp\. 69022–69042\.Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Changet al\.\(2024\)Y\. Chang, X\. Wang, J\. Wang, Y\. Wu, L\. Yang, K\. Zhu, H\. Chen, X\. Yi, C\. Wang, Y\. Wang,et al\.A survey on evaluation of large language models\.ACM transactions on intelligent systems and technology15\(3\),pp\. 1–45\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1)\.
- Chatterjeeet al\.\(2024\)A\. Chatterjee, H\. S\. V\. N\. S\. K\. Renduchintala, S\. Bhatia, and T\. ChakrabortyPOSIX: a prompt sensitivity index for large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 14550–14565\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.852/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.852)Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1),[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Chenet al\.\(2024\)L\. Chen, S\. Lou, B\. Huang, and Q\. ZhangDefining and extracting generalizable interaction primitives from DNNs\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OCqyFVFNeF)Cited by:[Appendix J](https://arxiv.org/html/2608.18539#A10.p3.1),[§1](https://arxiv.org/html/2608.18539#S1.p2.1),[§2](https://arxiv.org/html/2608.18539#S2.p2.1),[§4\.3](https://arxiv.org/html/2608.18539#S4.SS3.p8.1)\.
- Chenget al\.\(2025\)L\. Cheng, J\. Zhang, Q\. Ren, and Q\. ZhangRevisiting generalization power of a DNN in terms of symbolic interactions\.arXiv preprint arXiv:2502\.10162\.Cited by:[§E\.4](https://arxiv.org/html/2608.18539#A5.SS4.p1.1),[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1),[§4\.3](https://arxiv.org/html/2608.18539#S4.SS3.p8.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Conoveret al\.\(2023\)M\. Conover, M\. Hayes, A\. Mathur, J\. Xie, J\. Wan, S\. Shah, A\. Ghodsi, P\. Wendell, M\. Zaharia, and R\. XinFree dolly: introducing the world’s first truly open instruction\-tuned llm\(Website\)External Links:[Link](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm)Cited by:[§4\.3](https://arxiv.org/html/2608.18539#S4.SS3.p6.1)\.
- Covertet al\.\(2021\)I\. Covert, S\. Lundberg, and S\. LeeExplaining by removing: a unified framework for model explanation\.Journal of Machine Learning Research22\(209\),pp\. 1–90\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Dabkowski and Gal \(2017\)P\. Dabkowski and Y\. GalReal time image saliency for black box classifiers\.Advances in neural information processing systems30\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Denget al\.\(2024\)H\. Deng, N\. Zou, M\. Du, W\. Chen, G\. Feng, Z\. Yang, Z\. Li, and Q\. ZhangUnifying fourteen post\-hoc attribution methods with Taylor interactions\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(7\),pp\. 4625–4640\.Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Erricaet al\.\(2025\)F\. Errica, D\. Sanvito, G\. Siracusano, and R\. BifulcoWhat did I do wrong? quantifying LLMs’ sensitivity and consistency to prompt engineering\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1543–1558\.External Links:[Link](https://aclanthology.org/2025.naacl-long.73/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.73),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1),[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Fong and Vedaldi \(2017\)R\. C\. Fong and A\. VedaldiInterpretable explanations of black boxes by meaningful perturbation\.InProceedings of the IEEE international conference on computer vision,pp\. 3429–3437\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Fonget al\.\(2019\)R\. Fong, M\. Patrick, and A\. VedaldiUnderstanding deep networks via extremal perturbations and smooth masks\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 2950–2958\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Groeneveldet al\.\(2024\)D\. Groeneveld, I\. Beltagy, E\. Walsh, A\. Bhagia, R\. Kinney, O\. Tafjord, A\. Jha, H\. Ivison, I\. Magnusson, Y\. Wang,et al\.OLMo: accelerating the science of language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15789–15809\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. Singh Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. Renard Lavaud, M\. Lachaux, P\. Stock, T\. Le Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. El SayedMistral 7B\.arXiv e\-prints,pp\. arXiv:2310\.06825\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.06825),2310\.06825Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Kanget al\.\(2025\)J\. S\. Kang, L\. Butler, A\. Agarwal, Y\. E\. Erginbas, R\. Pedarsani, B\. Yu, and K\. RamchandranSPEX: scaling feature interaction explanations for LLMs\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=pRlKbAwczl)Cited by:[Appendix J](https://arxiv.org/html/2608.18539#A10.p6.1),[§4\.3](https://arxiv.org/html/2608.18539#S4.SS3.p8.1)\.
- Li and Zhang \(2023\)M\. Li and Q\. ZhangDoes a neural network really encode symbolic concepts?\.InInternational conference on machine learning,pp\. 20452–20469\.Cited by:[Appendix D](https://arxiv.org/html/2608.18539#A4.p1.1),[§F\.8\.2](https://arxiv.org/html/2608.18539#A6.SS8.SSS2.p1.1),[§1](https://arxiv.org/html/2608.18539#S1.p3.1),[§2](https://arxiv.org/html/2608.18539#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.18539#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.18539#S3.SS1.p2.1),[§3\.2](https://arxiv.org/html/2608.18539#S3.SS2.p2.1)\.
- Liet al\.\(2025\)Q\. Li, J\. Ruan, F\. Wu, Y\. Chen, Z\. Wei, and W\. ShenA unified approach to interpreting self\-supervised pre\-training methods for 3D point clouds via interactions\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 27315–27324\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52734.2025.02544)Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Luet al\.\(2024\)S\. Lu, H\. Schuff, and I\. GurevychHow are prompts different in terms of sensitivity?\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 5833–5856\.External Links:[Link](https://aclanthology.org/2024.naacl-long.325/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.325)Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1),[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Lundberg and Lee \(2017\)S\. M\. Lundberg and S\. LeeA unified approach to interpreting model predictions\.Advances in neural information processing systems30\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Muennighoffet al\.\(2024\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, P\. Walsh, O\. Tafjord, N\. Lambert,et al\.OLMoE: open mixture\-of\-experts language models\.arXiv preprint arXiv:2409\.02060\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- OLMoet al\.\(2025\)T\. OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, A\. Ettinger, M\. Guerquin, D\. Heineman, H\. Ivison, P\. W\. Koh, J\. Liu, S\. Malik, W\. Merrill, L\. J\. V\. Miranda, J\. Morrison, T\. Murray, C\. Nam, J\. Poznanski, V\. Pyatkin, A\. Rangapur, M\. Schmitz, S\. Skjonsberg, D\. Wadden, C\. Wilhelm, M\. Wilson, L\. Zettlemoyer, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi2 olmo 2 furious\.External Links:2501\.00656,[Link](https://arxiv.org/abs/2501.00656)Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§K\.1](https://arxiv.org/html/2608.18539#A11.SS1.p1.1)\.
- Quet al\.\(2024\)X\. Qu, D\. Dong, X\. Hu, T\. Zhu, W\. Sun, and Y\. ChengLlama\-MoE v2: exploring sparsity of Llama from perspective of mixture\-of\-experts with post\-training\.arXiv preprint arXiv:2411\.15708\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Razaviet al\.\(2025\)A\. Razavi, M\. Soltangheis, N\. Arabzadeh, S\. Salamat, M\. Zihayat, and E\. BagheriBenchmarking prompt sensitivity in large language models\.InEuropean Conference on Information Retrieval,pp\. 303–313\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1),[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Renet al\.\(2023a\)J\. Ren, M\. Li, Q\. Chen, H\. Deng, and Q\. ZhangDefining and quantifying the emergence of sparse concepts in DNNs\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 20280–20289\.Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.18539#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.18539#S3.SS1.p2.1)\.
- Renet al\.\(2025\)J\. Ren, X\. Zheng, J\. Liu, A\. Lizarraga, Y\. N\. Wu, L\. Lin, and Q\. ZhangMonitoring primitive interactions during the training of DNNs\.Proceedings of the AAAI Conference on Artificial Intelligence39\(19\),pp\. 20183–20191\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/34223),[Document](https://dx.doi.org/10.1609/aaai.v39i19.34223)Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p2.1),[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Renet al\.\(2023b\)Q\. Ren, H\. Deng, Y\. Chen, S\. Lou, and Q\. ZhangBayesian neural networks avoid encoding complex and perturbation\-sensitive concepts\.InInternational Conference on Machine Learning,pp\. 28889–28913\.Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Renet al\.\(2024a\)Q\. Ren, J\. Gao, W\. Shen, and Q\. ZhangWhere we have arrived in proving the emergence of sparse interaction primitives in DNNs\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3pWSL8My6B)Cited by:[§B\.2](https://arxiv.org/html/2608.18539#A2.SS2.p2.1.1),[§2](https://arxiv.org/html/2608.18539#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.18539#S3.SS1.p2.1)\.
- Renet al\.\(2024b\)Q\. Ren, J\. Zhang, Y\. Xu, Y\. Xin, D\. Liu, and Q\. ZhangTowards the dynamics of a DNN learning symbolic interactions\.Advances in Neural Information Processing Systems37,pp\. 50653–50688\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p2.1),[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Sclaret al\.\(2024\)M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. SuhrQuantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RIu5lyNXjT)Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1),[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Shenet al\.\(2024\)W\. Shen, Z\. Wei, Q\. Ren, B\. Zhang, S\. Huang, J\. Fan, and Q\. ZhangInterpretable rotation\-equivariant quaternion neural networks for 3D point cloud processing\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(5\),pp\. 3290–3304\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2023.3346383)Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Srivastavaet al\.\(2023\)A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso,et al\.Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on machine learning research\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1)\.
- Sturmfelset al\.\(2020\)P\. Sturmfels, S\. Lundberg, and S\. LeeVisualizing the impact of feature attribution baselines\.Distill5\(1\),pp\. e22\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Sunet al\.\(2024\)J\. Sun, C\. Shaib, and B\. C\. WallaceEvaluating the zero\-shot robustness of instruction\-tuned language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=g9diuvxN6D)Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Sundararajanet al\.\(2017\)M\. Sundararajan, A\. Taly, and Q\. YanAxiomatic attribution for deep networks\.InInternational conference on machine learning,pp\. 3319–3328\.Cited by:[Appendix A](https://arxiv.org/html/2608.18539#A1.p1.1)\.
- Tenneyet al\.\(2020\)I\. Tenney, J\. Wexler, J\. Bastings, T\. Bolukbasi, A\. Coenen, S\. Gehrmann, E\. Jiang, M\. Pushkarna, C\. Radebaugh, E\. Reif,et al\.The language interpretability tool: extensible, interactive visualizations and analysis for nlp models\.InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations,pp\. 107–118\.Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Wanget al\.\(2020\)Y\. Wang, Q\. Yao, J\. T\. Kwok, and L\. M\. NiGeneralizing from a few examples: a survey on few\-shot learning\.ACM computing surveys \(csur\)53\(3\),pp\. 1–34\.Cited by:[§4\.2](https://arxiv.org/html/2608.18539#S4.SS2.p5.1)\.
- Wenet al\.\(2026\)L\. Wen, L\. Zheng, H\. Li, L\. Sun, Z\. Wei, and W\. ShenInterpreting arithmetic reasoning in large language models using game\-theoretic interactions\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tRvzEL64dY)Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Yang, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, X\. Liu, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Guo, and Z\. FanQwen2 technical report\.External Links:2407\.10671,[Link](https://arxiv.org/abs/2407.10671)Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Zhanget al\.\(2020\)Q\. Zhang, X\. Wang, R\. Cao, Y\. N\. Wu, F\. Shi, and S\. ZhuExtraction of an explanatory graph to interpret a CNN\.IEEE transactions on pattern analysis and machine intelligence43\(11\),pp\. 3863–3877\.Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p2.1)\.
- Zhouet al\.\(2024\)H\. Zhou, H\. Zhang, H\. Deng, D\. Liu, W\. Shen, S\. Chan, and Q\. ZhangExplaining generalization power of a DNN using interactive concepts\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 17105–17113\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p2.1),[§2](https://arxiv.org/html/2608.18539#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.18539#S3.SS1.p2.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§1](https://arxiv.org/html/2608.18539#S1.p1.1)\.
- Zhuet al\.\(2024a\)K\. Zhu, J\. Wang, J\. Zhou, Z\. Wang, H\. Chen, Y\. Wang, L\. Yang, W\. Ye, Y\. Zhang, N\. Gong, and X\. XiePromptRobust: towards evaluating the robustness of large language models on adversarial prompts\.InProceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis,LAMPS ’24,New York, NY, USA,pp\. 57–68\.External Links:ISBN 9798400712098,[Link](https://doi.org/10.1145/3689217.3690621),[Document](https://dx.doi.org/10.1145/3689217.3690621)Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.
- Zhuet al\.\(2024b\)T\. Zhu, X\. Qu, D\. Dong, J\. Ruan, J\. Tong, C\. He, and Y\. ChengLlama\-MoE: building mixture\-of\-experts from Llama with continual pre\-training\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15913–15923\.Cited by:[§4\.1](https://arxiv.org/html/2608.18539#S4.SS1.p2.1)\.
- Zhuoet al\.\(2024\)J\. Zhuo, S\. Zhang, X\. Fang, H\. Duan, D\. Lin, and K\. ChenProSA: assessing and understanding the prompt sensitivity of LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1950–1976\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.108/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.108)Cited by:[§2](https://arxiv.org/html/2608.18539#S2.p1.1)\.

## Appendix AMasking Strategies of Input Variables

In attribution method research, it is common to employ a specific token or embedding to mask the input variables of a deep neural network \(DNN\)\([30](https://arxiv.org/html/2608.18539#bib.bib43);[2](https://arxiv.org/html/2608.18539#bib.bib44);[20](https://arxiv.org/html/2608.18539#bib.bib45)\)and use changes in network outputs on the masked samples to estimate attributions of different input variables\. The selection of a masking approach is complex, as each method has its weakness\. For example, replacing input variables with the mean baseline value \(the average of all samples\) or the zero baseline value can introduce out\-of\-distribution signals, thereby providing the model with artificial information, such as uniform grey or black dots in an image\([16](https://arxiv.org/html/2608.18539#bib.bib46);[2](https://arxiv.org/html/2608.18539#bib.bib44);[46](https://arxiv.org/html/2608.18539#bib.bib47)\)\. Additionally, blurring image pixels using a Gaussian kernel\([19](https://arxiv.org/html/2608.18539#bib.bib48);[20](https://arxiv.org/html/2608.18539#bib.bib45)\)as the masked state removes high\-frequency signals but fails to eliminate low\-frequency signals\([15](https://arxiv.org/html/2608.18539#bib.bib49);[44](https://arxiv.org/html/2608.18539#bib.bib50)\)\.

Given these challenges, we adopt a token replacement strategy, which is standard for the text domain\. This involves substituting the target input word with a dedicated\[MASK\]token at the embedding level\. For example, to mask the word ”green” in the input ”He is a green hand,” we would provide the LLM with the modified input ”He is a \[MASK\] hand\.” This approach effectively nullifies the specific semantic contribution of the target word without introducing out\-of\-distribution artifacts, ensuring a clean and consistent baseline for our interaction analysis\. For the specific\[MASK\]token for each LLM, please refer to Section 5 for details\.

## Appendix BProof of Theorem

### B\.1Proof of Universal Matching Property

In the main body of the paper, for the sake of simplicity and clarity, we introduced the Universal Matching Property \(Theorem 1\) primarily through the lens of AND interactions\. However, our empirical analysis and the underlying theoretical framework are built upon a more comprehensive AND\-OR interaction framework\. This extended framework, which incorporates both AND and OR interaction patterns, also adheres to the Universal Matching Property\.

In this section, we provide the formal proof for the Universal Matching Property of the complete AND\-OR interaction framework\. This proof is more general and naturally subsumes the proof for the AND interaction framework presented as Theorem 1 in the main text\. We will demonstrate that the output of the surrogate logical model, which is the sum of all AND\-OR interaction effects, can perfectly match the output of the Deep Neural Network \(DNN\) for any masked sample\.

Thesurrogate logical modelϕ⁡\(⋅\)\\phi\(\\cdot\)is defined as follows:

ϕ⁡\(𝒙T\)≜ϕ⁡\(𝒙∅\)\+∑S⊆N,S≠∅𝟙AND​\(S∣𝒙T\)⋅ISAND\+∑S⊆N,S≠∅𝟙OR​\(S∣𝒙T\)⋅ISOR,\\displaystyle\\phi\(\\bm\{x\}\_\{T\}\)\\triangleq\\phi\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\{\(S\\mid\\bm\{x\}\_\{T\}\)\}\\cdot I^\{\\text\{\\rm AND\}\}\_\{S\}\+\\sum\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\_\{\\text\{\\rm OR\}\}\{\(S\\mid\\bm\{x\}\_\{T\}\)\}\\cdot I^\{\\text\{\\rm OR\}\}\_\{S\},\(4\)where the AND trigger function𝟙AND​\(S∣𝒙T\)∈\{0,1\}\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)\\in\\\{0,1\\\}represents anAND relationshipbetween input variables inSS, which can also be termedAND interaction pattern; the OR trigger function𝟙OR​\(S∣𝒙T\)∈\{0,1\}\\mathds\{1\}\_\{\\text\{\\rm OR\}\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)\\in\\\{0,1\\\}represents anOR relationshipbetween input variables inSS, which can also be termedOR interaction pattern\. The scalar weightISANDI^\{\\text\{\\rm AND\}\}\_\{S\}quantifies the effect of an AND relationship, which can also be termedAND interaction effect; the scalar weightISORI^\{\\text\{\\rm OR\}\}\_\{S\}quantifies the effect of an OR relationship, which can also be termedOR interaction effect\. An AND relationship is activated only by the joint presence of all input variables in the setSS,i\.e\., all input variables inSSare not masked\. For instance, given the input sentence𝒙=“He is a green hand,”\\bm\{x\}=\\textit\{\`\`He\\ is\\ a\\ green\\ hand,''\}the co\-occurrence of the input variables in the setS=\{green,hand\}S=\\\{\\textit\{green\},\\textit\{hand\}\\\}contributes a numerical effectISANDI^\{\\text\{\\rm AND\}\}\_\{S\}that pushes the surrogate logical model’s inference towards the semantic meaning of“beginner\.”If an AND interactionSSis triggered,i\.e\.,𝟙AND​\(S∣𝒙T\)=1\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)=1, the corresponding interaction effectISANDI^\{\\text\{\\rm AND\}\}\_\{S\}is added to the output of the logical model\. Otherwise, if any word inSSis masked and the AND interaction is not triggered,i\.e\.,𝟙AND​\(S∣𝒙T\)=0\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)=0, the interaction effectISANDI^\{\\text\{\\rm AND\}\}\_\{S\}is not added to the output of the logical model\. An OR relationship is activated by the presence of any of all input variables in the setSS,i\.e\., any input variables inSSare not masked\. For instance, given the input sentence𝒙=“The service was terrible and the food was awful,”\\bm\{x\}=\\textit\{\`\`The\\ service\\ was\\ terrible\\ and\\ the\\ food\\ was\\ awful,''\}the presence of any input variables in the setS=\{terrible,awful\}S=\\\{\\textit\{terrible\},\\textit\{awful\}\\\}contributes a numerical effectISORI^\{\\text\{\\rm OR\}\}\_\{S\}that pushes the surrogate logical model’s inference towards a negative sentiment classification\. If an OR interactionSSis triggered,i\.e\.,𝟙OR​\(S∣𝒙T\)=1\\mathds\{1\}\_\{\\text\{\\rm OR\}\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)=1, the corresponding interaction effectISORI^\{\\text\{\\rm OR\}\}\_\{S\}is added to the output of the logical model\. Otherwise, if all words inSSare masked and the OR interaction is not triggered,i\.e\.,𝟙OR​\(S∣𝒙T\)=0\\mathds\{1\}\_\{\\text\{\\rm OR\}\}\(\{S\\mid\\bm\{x\}\_\{T\}\}\)=0, the interaction effectISORI^\{\\text\{\\rm OR\}\}\_\{S\}is not added to the output of the logical model\.𝒙∅\\bm\{x\}\_\{\\emptyset\}represents that all input variables inNNare masked\.

Definition of universal matching property for AND\-OR interactions\.When the scalar weights in the surrogate logical modelϕ⁡\(⋅\)\\phi\(\\cdot\)are set toISAND=∑T⊆S\(−1\)\|S\|−\|T\|​vand​\(𝒙T\)I^\{\\text\{\\rm AND\}\}\_\{S\}=\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)andISOR=−∑T⊆S\(−1\)\|S\|−\|T\|vor\(𝒙N∖T\)I^\{\\text\{\\rm OR\}\}\_\{S\}=\-\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus T\}\), the output ofϕ⁡\(⋅\)\\phi\(\\cdot\)can always match the output score of the DNNv⁡\(⋅\)v\(\\cdot\),i\.e\.,∀T⊆N,v⁡\(𝒙T\)=ϕ⁡\(𝒙T\)\\forall T\\subseteq N,v\(\\bm\{x\}\_\{T\}\)=\\phi\(\\bm\{x\}\_\{T\}\)\. Herevand​\(𝒙T\)\+vor​\(𝒙T\)=v⁡\(𝒙T\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\+v\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)=v\(\\bm\{x\}\_\{T\}\)\.

We need to prove that given an input sample𝒙\\bm\{x\}, for each masked sample\{𝒙T\|T⊆N\}\\\{\\bm\{x\}\_\{T\}\|T\\subseteq N\\\}, the network output scorev⁡\(𝒙T\)∈ℝv\(\\bm\{x\}\_\{T\}\)\\in\\mathbb\{R\}can be well matched by the surrogate logical modelϕ⁡\(𝒙T\)\\phi\(\\bm\{x\}\_\{T\}\)\. The surrogate logical modelϕ⁡\(𝒙T\)\\phi\(\\bm\{x\}\_\{T\}\)uses the sum of AND interactions and OR interactions to accurately explain/match the network output scorev⁡\(𝒙T\)v\(\\bm\{x\}\_\{T\}\)\.

∀T⊆N,v\(𝒙T\)=ϕ\(𝒙T\)\.\\displaystyle\\forall T\\subseteq N,v\(\\bm\{x\}\_\{T\}\)=\\phi\(\\bm\{x\}\_\{T\}\)\.\(5\)ϕ⁡\(𝒙T\)=ϕ⁡\(𝒙∅\)\+∑S⊆N,S≠∅𝟙AND​\(S∣𝒙T\)⋅ISAND\+∑S⊆N,S≠∅𝟙OR​\(S∣𝒙T\)⋅ISOR,\\displaystyle\\phi\(\\bm\{x\}\_\{T\}\)=\\phi\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\_\{\\text\{\\rm AND\}\}\{\(S\\mid\\bm\{x\}\_\{T\}\)\}\\cdot I^\{\\text\{\\rm AND\}\}\_\{S\}\+\\sum\_\{S\\subseteq N,S\\neq\\emptyset\}\\mathds\{1\}\_\{\\text\{\\rm OR\}\}\{\(S\\mid\\bm\{x\}\_\{T\}\)\}\\cdot I^\{\\text\{\\rm OR\}\}\_\{S\},=v⁡\(𝒙∅\)\+∑S⊆T,S≠∅ISAND⏟vand​\(𝒙T\)\+∑S⊆N,S∩T≠∅ISOR⏟vor​\(𝒙T\)\\displaystyle=\\underbrace\{v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm AND\}\}\_\{S\}\}\_\{v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)\}\+\\underbrace\{\\sum\\nolimits\_\{S\\subseteq N,S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}\}\_\{v\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)\}
###### Proof\.

\(1\) Universal matching property of AND interactions\.For all2n2^\{n\}masked samples\{𝒙T∣T⊆N\}\\\{\\bm\{x\}\_\{T\}\\mid T\\subseteq N\\\}, what we need to prove is that the outputvand​\(𝒙T\)v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)of a DNN can be universally explained by all the interactions inT⊆NT\\subseteq N,i\.e\.,∀S⊆T,S≠∅,vand​\(𝒙T\)=∑S⊆T,S≠∅ISAND​\(𝒙\)=v⁡\(𝒙∅\)\+∑S⊆T,S≠∅ISAND\\forall S\\subseteq T,S\\neq\\emptyset,v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\. Here,v⁡\(𝒙∅\)=vand​\(𝒙∅\)v\(\\bm\{x\}\_\{\\emptyset\}\)=v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)\.

According to the definition of the AND interaction,ISAND​\(𝒙\)=∑L⊆S\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\. To simplify the computation of the sum of AND interactions∑S⊆T,S≠∅ISAND​\(𝒙\)=∑S⊆T,S≠∅∑L⊆S\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\), we exchange the order of summation of the setL⊆S⊆TL\\subseteq S\\subseteq Tand the setS⊇LS\\supseteq L\. Given a set of input variablesLL, we compute all linear combinations of all setsSScontainingLLwith respect to the model outputsvand​\(𝒙S\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{S\}\),i\.e\.,∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|vand\(𝒙L\)\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\. Then, we compute all summations over the setL⊆TL\\subseteq Tas∑S⊆T,S≠∅ISAND\(𝒙\)=∑L⊆T∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|vand\(𝒙L\)\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{L\\subseteq T\}\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\. Then, we can compute different cases ofL⊆S⊆TL\\subseteq S\\subseteq Tas follows:

\(1\) WhenL=T=SL=T=S,∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|vand\(𝒙L\)=\(−1\)\|T\|−\|T\|vand\(𝒙L\)=vand\(𝒙L\)\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=\(\-1\)^\{\|T\|\-\|T\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\.

\(2\) WhenL⊆S⊆T,L≠TL\\subseteq S\\subseteq T,L\\neq T, let us consider the linear combinations of all setsSSwith number\|S\|\|S\|for the model outputvand​\(𝒙L\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\), respectively\. Letm:=\|S\|−\|L\|m:=\|S\|\-\|L\|, \(0≤m≤\|T\|−\|L\|0\\leq m\\leq\|T\|\-\|L\|\), then there are a total ofC\|T\|−\|L\|mC\_\{\|T\|\-\|L\|\}^\{m\}combinations of all setsSSof order\|S\|\|S\|\. GivenLL, accumulating the model outputsvand​\(𝒙L\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)corresponding to allS⊇LS\\supseteq L, we can get∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|vand\(𝒙L\)=vand\(𝒙L\)⋅∑m=0\|T\|−\|L\|C\|T\|−\|L\|m​\(−1\)m⏟=0=0\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\\cdot\\underbrace\{\\sum\\nolimits\_\{m=0\}^\{\|T\|\-\|L\|\}C\_\{\|T\|\-\|L\|\}^\{m\}\(\-1\)^\{m\}\}\_\{=0\}=0\.

Considering all the cases, the complete derivation of the sum of AND interactions is as follows\.

\\displaystyle∑S⊆T,S≠∅ISAND\\displaystyle\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\(6\)=\\displaystyle=∑S⊆T,S≠∅∑L⊆S\(−1\)\|S\|−\|L\|​vand​\(𝒙L\)\\displaystyle\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)=\\displaystyle=∑L⊆T∑S:L⊆S⊆T\(−1\)\|S\|−\|L\|vand\(𝒙L\)−vand\(𝒙∅\)\\displaystyle\\sum\\nolimits\_\{L\\subseteq T\}\\sum\\nolimits\_\{S:L\\subseteq S\\subseteq T\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\-v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=\\displaystyle=vand​\(𝒙T\)⏟L=T\+∑L⊆T,L≠Tvand​\(𝒙L\)⋅∑m=0\|T\|−\|L\|C\|T\|−\|L\|m​\(−1\)m⏟=0−vand​\(𝒙∅\)\\displaystyle\\underbrace\{v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\}\_\{L=T\}\+\\sum\\nolimits\_\{L\\subseteq T,L\\neq T\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{L\}\)\\cdot\\underbrace\{\\sum\\nolimits\_\{m=0\}^\{\|T\|\-\|L\|\}C\_\{\|T\|\-\|L\|\}^\{m\}\(\-1\)^\{m\}\}\_\{=0\}\-v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=\\displaystyle=vand​\(𝒙T\)−v⁡\(𝒙∅\)\\displaystyle v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\-v\(\\bm\{x\}\_\{\\emptyset\}\)
Therefore, we have proven that∀∅≠T⊆N,vand​\(𝒙T\)=v⁡\(𝒙∅\)\+∑S⊆T,S≠∅ISAND\\forall\\emptyset\\neq T\\subseteq N,v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I\_\{S\}^\{\\text\{\\rm AND\}\}\.

\(2\) Universal matching theorem of OR interactions\.What we need to prove is that∀T⊆N,vor\(𝒙T\)=∑S∈\{S:S∩T≠∅\}∪\{∅\}ISOR=∑S:S∩T≠∅ISOR\\forall T\\subseteq N,v\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\\nolimits\_\{S\\in\\\{S:S\\cap T\\neq\\emptyset\\\}\\cup\\\{\\emptyset\\\}\}I^\{\\text\{\\rm OR\}\}\_\{S\}=\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}\. HereI∅OR=vor​\(𝒙∅\)=0I^\{\\text\{\\rm OR\}\}\_\{\\emptyset\}=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=0\.

According to the definition of the OR interaction,ISOR:=−∑L⊆S\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)I^\{\\text\{\\rm OR\}\}\_\{S\}:=\-\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\. To simplify the computation of the sum of OR interactions∑S:S∩T≠∅ISOR=∑S:S∩T≠∅\[−∑L⊆S\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)\]\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}=\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}\\left\[\-\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\\right\], we also exchange the order of summation of the setL⊆S⊆NL\\subseteq S\\subseteq Nand the setS:S∩T=∅S:S\\cap T=\\emptyset\. Given a set of input variablesLL, we compute all linear combinations of all setsSScontainingLLwith respect to the model outputsvor​\(𝒙N∖L\)v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\),i\.e\.,∑S:S∩T≠∅,N⊇S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,N\\supseteq S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\. Then, we compute all summations over the setL⊆NL\\subseteq Nas∑S:S∩T≠∅ISOR=−∑L⊆N∑S:S∩T≠∅,N⊇S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}=\-\\sum\\nolimits\_\{L\\subseteq N\}\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,N\\supseteq S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\. Then, we can compute different cases ofL⊆S⊆N,S∩T≠∅L\\subseteq S\\subseteq N,S\\cap T\\neq\\emptysetas follows:

\(1\) WhenL=NL=N\(thenS=NS=N\),∑S:S∩T≠∅,S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)=\(−1\)\|N\|−\|N\|vor\(𝒙∅\)=vor\(𝒙∅\)=0\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=\(\-1\)^\{\|N\|\-\|N\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=0, HereI∅OR=vor​\(𝒙∅\)=0I^\{\\text\{\\rm OR\}\}\_\{\\emptyset\}=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=0\.

\(2\) WhenL=N∖TL=N\\setminus T, for all setsS:S⊇L,S∩T≠∅S:S\\supseteq L,S\\cap T\\neq\\emptyset\(thenS≠N∖T,S≠LS\\neq N\\setminus T,S\\neq L\), let us consider the linear combinations of all setsSSwith number\|S\|\|S\|for the model outputvor​\(𝒙T\)v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\), respectively\. Let\|S′\|:=\|S\|−\|L\|\|S^\{\\prime\}\|:=\|S\|\-\|L\|, \(1≤\|S′\|≤\|T\|1\\leq\|S^\{\\prime\}\|\\leq\|T\|\), then there are a total ofC\|T\|\|S′\|C\_\{\|T\|\}^\{\|S^\{\\prime\}\|\}combinations of all setsSSof order\|S\|\|S\|\. Thus,∑S:S∩T≠∅,S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)=vor\(𝒙T\)⋅∑\|S′\|=1\|T\|C\|T\|\|S′\|​\(−1\)\|S′\|⏟=−1=−vor\(𝒙T\)\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\\cdot\\underbrace\{\\sum\\nolimits\_\{\|S^\{\\prime\}\|=1\}^\{\|T\|\}C\_\{\|T\|\}^\{\|S^\{\\prime\}\|\}\(\-1\)^\{\|S^\{\\prime\}\|\}\}\_\{=\-1\}=\-v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\.

\(3\) WhenL∩T≠∅,L≠NL\\cap T\\neq\\emptyset,L\\neq N, for all setsS:S⊇L,S∩T≠∅S:S\\supseteq L,S\\cap T\\neq\\emptyset, let us consider the linear combinations of all setsSSwith number\|S\|\|S\|for the model outputvor​\(𝒙T\)v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\), respectively\. Let us split\|S\|−\|L\|\|S\|\-\|L\|into\|S′\|\|S^\{\\prime\}\|and\|S′′\|\|S^\{\\prime\\prime\}\|,i\.e\.,\|S\|−\|L\|=\|S′\|\+\|S′′\|\|S\|\-\|L\|=\|S^\{\\prime\}\|\+\|S^\{\\prime\\prime\}\|, whereS′=\{i\|i∈S,i∉L,i∈N∖T\}S^\{\\prime\}=\\\{i\|i\\in S,i\\notin L,i\\in N\\setminus T\\\},S′′=\{i\|i∈S,i∉L,i∈T\}S^\{\\prime\\prime\}=\\\{i\|i\\in S,i\\notin L,i\\in T\\\}\(then0≤\|S′′\|≤\|T\|−\|T∩L\|0\\leq\|S^\{\\prime\\prime\}\|\\leq\|T\|\-\|T\\cap L\|\) andS′\+S′′\+L=SS^\{\\prime\}\+S^\{\\prime\\prime\}\+L=S\. Thus, there are a total ofC\|T\|−\|T∩L\|\|S′′\|C\_\{\|T\|\-\|T\\cap L\|\}^\{\|S^\{\\prime\\prime\}\|\}combinations of all setsS′′S^\{\\prime\\prime\}of order\|S′′\|\|S^\{\\prime\\prime\}\|\. Thus,∑S:S∩T≠∅,S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)=vor\(𝒙N∖L\)⋅∑S′⊆N∖T∖L∑\|S′′\|=0\|T\|−\|T∩L\|C\|T\|−\|T∩L\|\|S′′\|​\(−1\)\|S′\|\+\|S′′\|⏟=0=0\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\\cdot\\sum\_\{S^\{\\prime\}\\subseteq N\\setminus T\\setminus L\}\\underbrace\{\\sum\\nolimits\_\{\|S^\{\\prime\\prime\}\|=0\}^\{\|T\|\-\|T\\cap L\|\}C\_\{\|T\|\-\|T\\cap L\|\}^\{\|S^\{\\prime\\prime\}\|\}\(\-1\)^\{\|S^\{\\prime\}\|\+\|S^\{\\prime\\prime\}\|\}\}\_\{=0\}=0\.

\(4\) WhenL∩T=∅,L≠N∖TL\\cap T=\\emptyset,L\\neq N\\setminus T, let us split\|S\|−\|L\|\|S\|\-\|L\|into\|S′\|\|S^\{\\prime\}\|and\|S′′\|\|S^\{\\prime\\prime\}\|,i\.e\.,\|S\|−\|L\|=\|S′\|\+\|S′′\|\|S\|\-\|L\|=\|S^\{\\prime\}\|\+\|S^\{\\prime\\prime\}\|, whereS′=\{i\|i∈S,i∉L,i∈N∖T\}S^\{\\prime\}=\\\{i\|i\\in S,i\\notin L,i\\in N\\setminus T\\\},S′′=\{i\|i∈S,i∈T\}S^\{\\prime\\prime\}=\\\{i\|i\\in S,i\\in T\\\}\(then0≤\|S′′\|≤\|T\|0\\leq\|S^\{\\prime\\prime\}\|\\leq\|T\|\) andS′\+S′′\+L=SS^\{\\prime\}\+S^\{\\prime\\prime\}\+L=S\. Thus, there are a total ofC\|T\|\|S′′\|C\_\{\|T\|\}^\{\|S^\{\\prime\\prime\}\|\}combinations of all setsS′′S^\{\\prime\\prime\}of order\|S′′\|\|S^\{\\prime\\prime\}\|\. Thus,∑S:S∩T≠∅,S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)=vor\(𝒙N∖L\)⋅∑S′⊆N∖T∖L∑\|S′′\|=0\|T\|C\|T\|\|S′′\|​\(−1\)\|S′\|\+\|S′′\|⏟=0=0\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\\cdot\\sum\_\{S^\{\\prime\}\\subseteq N\\setminus T\\setminus L\}\\underbrace\{\\sum\\nolimits\_\{\|S^\{\\prime\\prime\}\|=0\}^\{\|T\|\}C\_\{\|T\|\}^\{\|S^\{\\prime\\prime\}\|\}\(\-1\)^\{\|S^\{\\prime\}\|\+\|S^\{\\prime\\prime\}\|\}\}\_\{=0\}=0\.

Considering all the cases, the complete derivation of the sum of OR interactions is as follows\.

∑S:S∩T≠∅IORS\\displaystyle\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}=∑S:S∩T≠∅\[−∑L⊆S\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)\]\\displaystyle=\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}\\left\[\-\\sum\\nolimits\_\{L\\subseteq S\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)\\right\]\(7\)=−∑L⊆N∑S:S∩T≠∅,N⊇S⊇L\(−1\)\|S\|−\|L\|vor\(𝒙N∖L\)\\displaystyle=\-\\sum\\nolimits\_\{L\\subseteq N\}\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset,N\\supseteq S\\supseteq L\}\(\-1\)^\{\|S\|\-\|L\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=−\[∑\|S′\|=1\|T\|C\|T\|\|S′\|\(−1\)\|S′\|\]⋅vor​\(𝒙T\)⏟L=N∖T−vor​\(𝒙∅\)⏟L=N\\displaystyle=\-\\left\[\\sum\_\{\|S^\{\\prime\}\|=1\}^\{\|T\|\}C\_\{\|T\|\}^\{\|S^\{\\prime\}\|\}\(\-1\)^\{\|S^\{\\prime\}\|\}\\right\]\\cdot\\underbrace\{v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\}\_\{L=N\\setminus T\}\-\\underbrace\{v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)\}\_\{L=N\}−∑L∩T≠∅,L≠N\[∑S′⊆N∖T∖L\(∑\|S′′\|=0\|T\|−\|T∩L\|C\|T\|−\|T∩L\|\|S′′\|\(−1\)\|S′\|\+\|S′′\|\)\]⋅vor\(𝒙N∖L\)\\displaystyle\-\\sum\_\{L\\cap T\\neq\\emptyset,L\\neq N\}\\left\[\\sum\_\{S^\{\\prime\}\\subseteq N\\setminus T\\setminus L\}\\left\(\\sum\_\{\|S^\{\\prime\\prime\}\|=0\}^\{\|T\|\-\|T\\cap L\|\}C\_\{\|T\|\-\|T\\cap L\|\}^\{\|S^\{\\prime\\prime\}\|\}\(\-1\)^\{\|S^\{\\prime\}\|\+\|S^\{\\prime\\prime\}\|\}\\right\)\\right\]\\cdot v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)−∑L∩T=∅,L≠N∖T\[∑S′⊆N∖T∖L\(∑\|S′′\|=0\|T\|C\|T\|\|S′′\|\(−1\)\|S′\|\+\|S′′\|\)\]⋅vor\(𝒙N∖L\)\\displaystyle\-\\sum\_\{L\\cap T=\\emptyset,L\\neq N\\setminus T\}\\left\[\\sum\_\{S^\{\\prime\}\\subseteq N\\setminus T\\setminus L\}\\left\(\\sum\_\{\|S^\{\\prime\\prime\}\|=0\}^\{\|T\|\}C\_\{\|T\|\}^\{\|S^\{\\prime\\prime\}\|\}\(\-1\)^\{\|S^\{\\prime\}\|\+\|S^\{\\prime\\prime\}\|\}\\right\)\\right\]\\cdot v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=−\(−1\)⋅vor\(𝒙T\)−vor\(𝒙∅\)−∑L∩T≠∅,L≠N\[∑S′⊆N∖T∖L0\]⋅vor\(𝒙N∖L\)\\displaystyle=\-\(\-1\)\\cdot v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\-v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)\-\\sum\_\{L\\cap T\\neq\\emptyset,L\\neq N\}\\left\[\\sum\_\{S^\{\\prime\}\\subseteq N\\setminus T\\setminus L\}0\\right\]\\cdot v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)−∑L∩T=∅,L≠N∖T\[∑S′⊆N∖T∖L0\]⋅vor\(𝒙N∖L\)\\displaystyle\-\\sum\_\{L\\cap T=\\emptyset,L\\neq N\\setminus T\}\\left\[\\sum\_\{S^\{\\prime\}\\subseteq N\\setminus T\\setminus L\}0\\right\]\\cdot v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus L\}\)=vor​\(𝒙T\)−vor​\(𝒙∅\)\\displaystyle=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\-v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{\\emptyset\}\)=vor​\(𝒙T\)\\displaystyle=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)
Therefore, we have proven that∀T⊆N,vor\(𝒙T\)=∑S:S∩T≠∅ISOR\\forall T\\subseteq N,v\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\\nolimits\_\{S:S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}\.

\(3\) Universal matching theorem of AND\-OR interactions\.With the universal matching property of AND interactions and the universal matching property of OR interactions, we can easily getv⁡\(𝒙T\)=ϕ⁡\(𝒙T\)=vand​\(𝒙T\)\+vor​\(𝒙T\)=v⁡\(𝒙∅\)\+∑S⊆T,S≠∅ISAND\+∑S⊆N,S∩T≠∅ISORv\(\\bm\{x\}\_\{T\}\)=\\phi\(\\bm\{x\}\_\{T\}\)=v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)\+v\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm AND\}\}\_\{S\}\+\\sum\_\{S\\subseteq N,S\\cap T\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}, thus, we obtain the universal matching property of AND\-OR interactions\. ∎

### B\.2Proof of Sparsity Property

Given all the masked samples\{𝒙T∣T⊆N\}\\\{\\bm\{x\}\_\{T\}\\mid T\\subseteq N\\\}, the surrogate logical modelϕ⁡\(𝒙T\)\\phi\(\\bm\{x\}\_\{T\}\)only utilizes a small set of salient AND interactions inΩAND\\Omega^\{\\text\{\\rm AND\}\}and salient OR interactions inΩOR\\Omega^\{\\text\{\\rm OR\}\}to approximate the network output scorev⁡\(𝒙T\)v\(\\bm\{x\}\_\{T\}\)\. That is, the network’s output can be well approximated by a small set of AND\-OR interactions\.

v⁡\(𝒙T\)=ϕ⁡\(𝒙T\)≈v⁡\(𝒙∅\)\+∑S⊆T,S≠∅,S∈ΩANDISAND\+∑S⊆T,S≠∅,S∈ΩORISORv\(\\bm\{x\}\_\{T\}\)\\\!=\\\!\\phi\(\\bm\{x\}\_\{T\}\)\\\!\\approx\\\!v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega\_\{\\text\{AND\}\}\}I^\{\\text\{\\rm AND\}\}\_\{S\}\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega\_\{\\text\{OR\}\}\}I^\{\\text\{\\rm OR\}\}\_\{S\}\(8\)
###### Proof\.

It has been proven by[39](https://arxiv.org/html/2608.18539#bib.bib19)that under three common conditions777Here are the three conditions: \(1\) The DNN doesn’t encode extremely high\-order AND interactions\. \(2\) The DNN performs effectively on masked samples and exhibits greater confidence as the input sample is less masked\. \(3\) When we increase the number of masked input variables, the confidence of the DNN does not drop significantly\., the output scorevand​\(𝒙T\)v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)of a well\-trained DNN on all2n2^\{n\}masked samples\{𝒙T\|T⊆N\}\\\{\\bm\{x\}\_\{T\}\|T\\subseteq N\\\}could be universally estimated by a small number of AND interactionsT∈ΩANDT\\in\\Omega^\{\\text\{\\rm AND\}\}with salient interaction effectsISANDI^\{\\text\{\\rm AND\}\}\_\{S\},*s\.t\.*,\|ΩAND\|≪2n\|\\Omega^\{\\text\{\\rm AND\}\}\|\\ll 2^\{n\},i\.e\.,∀T⊆N,vand​\(𝒙T\)=∑S⊆T,S≠∅ISAND≈∑S⊆T,S≠∅,S∈ΩANDISAND\\forall T\\subseteq N,v\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm AND\}\}\_\{S\}\\approx\\sum\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega^\{\\text\{\\rm AND\}\}\}I^\{\\text\{\\rm AND\}\}\_\{S\}\. According to Eq\. \([6](https://arxiv.org/html/2608.18539#A2.E6)\),vand​\(𝒙T\)=v⁡\(𝒙∅\)\+∑S⊆T,S≠∅ISANDv\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)=v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm AND\}\}\_\{S\}\. Therefore,vand​\(𝒙T\)≈v⁡\(𝒙∅\)\+∑S⊆T,S≠∅,S∈ΩANDISANDv\_\{\\text\{\\rm and\}\}\(\\bm\{x\}\_\{T\}\)\\approx v\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega^\{\\text\{\\rm AND\}\}\}I^\{\\text\{\\rm AND\}\}\_\{S\}\.

Besides, as proven in Section[C](https://arxiv.org/html/2608.18539#A3), the OR interaction can be considered as a special AND interaction\. Thus, the confidence scorevor​\(𝒙T\)v\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)of a well\-trained DNN on all2n2^\{n\}masked samples\{𝒙T\|T⊆N\}\\\{\\bm\{x\}\_\{T\}\|T\\subseteq N\\\}could be universally estimated by a small number of OR interactionsT∈ΩORT\\in\\Omega^\{\\text\{\\rm OR\}\}with salient interaction effectsISORI^\{\\text\{\\rm OR\}\}\_\{S\},*s\.t\.*,\|ΩOR\|≪2n\|\\Omega^\{\\text\{\\rm OR\}\}\|\\ll 2^\{n\}\. Similarly,vor​\(𝒙T\)=∑S⊆T,S≠∅ISOR≈∑S⊆T,S≠∅,S∈ΩORISORv\_\{\\text\{\\rm or\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset\}I^\{\\text\{\\rm OR\}\}\_\{S\}\\approx\\sum\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega^\{\\text\{\\rm OR\}\}\}I^\{\\text\{\\rm OR\}\}\_\{S\}

Thus, for each randomly masked sample𝒙T,T⊆N\\bm\{x\}\_\{T\},T\\subseteq N, the surrogate logical modelϕ⁡\(𝒙T\)\\phi\(\\bm\{x\}\_\{T\}\)can use a small number of salient AND\-OR interactions to approximate the network output scorev⁡\(𝒙T\)v\(\\bm\{x\}\_\{T\}\),i\.e\.,v⁡\(𝒙T\)=ϕ⁡\(𝒙T\)=vand​\(𝒙T\)\+vor​\(𝒙T\)≈\(𝒙∅\)\+∑S⊆T,S≠∅,S∈ΩANDISAND\+∑S⊆T,S≠∅,S∈ΩORISORv\(\\bm\{x\}\_\{T\}\)\\\!=\\\!\\phi\(\\bm\{x\}\_\{T\}\)=v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\+v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\\\!\\approx\(\\bm\{x\}\_\{\\emptyset\}\)\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega\_\{\\text\{AND\}\}\}I^\{\\text\{\\rm AND\}\}\_\{S\}\+\\sum\\nolimits\_\{S\\subseteq T,S\\neq\\emptyset,S\\in\\Omega\_\{\\text\{OR\}\}\}I^\{\\text\{\\rm OR\}\}\_\{S\}\.

∎

## Appendix COR Interactions Can Be Considered as Special AND Interactions

If we reverse the definition of the masked state and the unmasked state of the input variable, the OR interactionISORI^\{\\text\{\\rm OR\}\}\_\{S\}can be considered as a special kind of AND interactionISANDI^\{\\text\{\\rm AND\}\}\_\{S\}\.

Given an input sample𝒙∈ℝn\\bm\{x\}\\in\\mathbb\{R\}^\{n\}and the output score of a DNN asv⁡\(⋅\)v\(\\cdot\), if we randomly mask input variables in𝒙\\bm\{x\}, we can get all2n2^\{n\}masked samples\. Let𝒙S\\bm\{x\}\_\{S\}denote the certain masked input sample when input variables inN∖SN\\setminus Sare all masked and input variables in S are kept unchanged\.

\(𝒙S\)i=\{xi,i∈Sbi,i∈N∖S\(\\bm\{x\}\_\{S\}\)\_\{i\}=\\begin\{cases\}x\_\{i\},&\\text\{$i\\in S$\}\\\\ b\_\{i\},&\\text\{$i\\in N\\setminus S$\}\\end\{cases\}\(9\)where𝐛∈ℝn\\mathbf\{b\}\\in\\mathbb\{R\}^\{n\}are baseline values to represent the masked state of input variables\.

If we reverse the definition of the masked state and the unmasked state of an input variable,i\.e\., we consider𝐛\\mathbf\{b\}as the input sample and consider𝒙\\bm\{x\}as the masked state, then the masked sample𝒙~S\\widetilde\{\\bm\{x\}\}\_\{S\}can be defined as follows\.

\(𝒙~S\)i=\{bi,i∈Sxi,i∈N∖S\(\\widetilde\{\\bm\{x\}\}\_\{S\}\)\_\{i\}=\\begin\{cases\}b\_\{i\},&\\text\{$i\\in S$\}\\\\ x\_\{i\},&\\text\{$i\\in N\\setminus S$\}\\end\{cases\}\(10\)Thus, we can get𝒙N∖S=𝒙~S\\bm\{x\}\_\{N\\setminus S\}=\\widetilde\{\\bm\{x\}\}\_\{S\}\. To simplify the analysis, let us assumevand​\(𝒙S\)=vor​\(𝒙S\)=0\.5​v​\(𝒙S\)v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{S\}\)=v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{S\}\)=0\.5v\(\\bm\{x\}\_\{S\}\), then the OR interactionISORI^\{\\text\{\\rm OR\}\}\_\{S\}can be regarded as a specific AND interactionISAND​\(𝒙~\)I^\{\\text\{AND\}\}\_\{S\}\(\\widetilde\{\\bm\{x\}\}\)as follows\.

ISOR​\(x\)=−∑T⊆S\(−1\)\|S\|−\|T\|vor\(𝒙N∖T\),=−∑T⊆S\(−1\)\|S\|−\|T\|vor\(𝒙~T\),=−∑T⊆S\(−1\)\|S\|−\|T\|vand\(𝒙~T\),=−ISAND​\(𝒙~\)\.\\begin\{split\}I^\{\\text\{\\rm OR\}\}\_\{S\}\(\\textbf\{x\}\)&=\-\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus T\}\),\\\\ &=\-\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{or\}\}\(\\widetilde\{\\bm\{x\}\}\_\{T\}\),\\\\ &=\-\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{and\}\}\(\\widetilde\{\\bm\{x\}\}\_\{T\}\),\\\\ &=\-I^\{\\text\{AND\}\}\_\{S\}\(\\widetilde\{\\bm\{x\}\}\)\.\\end\{split\}\(11\)Now we have proven that OR interactions can be considered as special AND interactions\.

## Appendix DDetails of Extracting the Sparsest AND\-OR Interactions

We follow[27](https://arxiv.org/html/2608.18539#bib.bib17)to extract AND\-OR interactions\. Given a masked sample𝒙T\\bm\{x\}\_\{T\}, the output score of the networkv⁡\(𝒙T\)v\(\\bm\{x\}\_\{T\}\)can be decomposed into a combination of AND interaction and OR interaction,i\.e\.,v⁡\(𝒙T\)=vand​\(𝒙T\)\+vor​\(𝒙T\)v\(\\bm\{x\}\_\{T\}\)=v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)\+v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)\. Specifically,vand​\(𝒙T\)=0\.5⋅v⁡\(𝒙T\)\+γTv\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)=0\.5\\cdot v\(\\bm\{x\}\_\{T\}\)\+\\gamma\_\{T\}andvor​\(𝒙T\)=0\.5⋅v⁡\(𝒙T\)−γTv\_\{\\text\{or\}\}\(\\bm\{x\}\_\{T\}\)=0\.5\\cdot v\(\\bm\{x\}\_\{T\}\)\-\\gamma\_\{T\}, where\{γT∣T⊆N\}\\\{\\gamma\_\{T\}\\mid T\\subseteq N\\\}is a set of learnable parameters\. The parameters\{γT\}\\\{\\gamma\_\{T\}\\\}were trained through minimizing the following LASSO\-like loss to obtain sparse interactions:

min⁡∑S⊆N\{γT\}⁡\|ISAND​\(𝒙\)\|\+\|ISOR​\(𝒙\)\|,\\min\_\{\\\{\\gamma\_\{T\}\\\}\}\\sum\_\{S\\subseteq N\}\|I^\{\\text\{AND\}\}\_\{S\}\(\\bm\{x\}\)\|\+\|I^\{\\text\{OR\}\}\_\{S\}\(\\bm\{x\}\)\|,\(12\)whereISAND​\(𝒙\)=∑T⊆S\(−1\)\|S\|−\|T\|​vand​\(𝒙T\)=∑T⊆S\(−1\)\|S\|−\|T\|​\(0\.5⋅v⁡\(𝒙T\)\+γT\)I^\{\\text\{\\rm AND\}\}\_\{S\}\(\\bm\{x\}\)=\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{and\}\}\(\\bm\{x\}\_\{T\}\)=\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}\(0\.5\\cdot v\(\\bm\{x\}\_\{T\}\)\+\\gamma\_\{T\}\)andISOR\(𝒙\)=−∑T⊆S\(−1\)\|S\|−\|T\|vor\(𝒙N∖T\)=−∑T⊆S\(−1\)\|S\|−\|T\|\(0\.5⋅v\(𝒙T\)−γT\)I^\{\\text\{\\rm OR\}\}\_\{S\}\(\\bm\{x\}\)=\-\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}v\_\{\\text\{or\}\}\(\\bm\{x\}\_\{N\\setminus T\}\)=\-\\sum\\nolimits\_\{T\\subseteq S\}\(\-1\)^\{\|S\|\-\|T\|\}\(0\.5\\cdot v\(\\bm\{x\}\_\{T\}\)\-\\gamma\_\{T\}\)\. Thus, we can extract the sparsest set of AND\-OR interactions\.

Table 1:A comprehensive list and characteristics of LLMs, grouped by model family and model series\.Model FamilyModel SeriesModelTypeArchitectureScaleLlama \(10 models\)Llama 2 \(6 models\)Llama\-2\-7bBaseDense7BLlama\-2\-7b\-chatChatDense7BLlama\-2\-13bBaseDense13BLlama\-2\-13b\-chatChatDense13BLlama\-2\-70bBaseDense70BLlama\-2\-70b\-chatChatDense70BLlama 3 \(2 models\)Llama\-3\-8bBaseDense8BLlama\-3\-8b\-instructInstructDense8BLlama MoE \(2 models\)Llama\-moe\-v1\-3\_5b\-2\_8\-sftInstructMoE3\.5B \(Activated\)Llama\-moe\-v2\-3\_8b\-2\_8\-sftInstructMoE3\.8B \(Activated\)Mistral \(4 models\)Mistral \(2 models\)Mistral\-7b\-v0\.3BaseDense7BMistral\-7b\-v0\.3\-instructInstructDense7BMixtral \(2 models\)Mixtral\-8x7bBaseMoE13B \(Activated\)Mixtral\-8x7b\-instructInstructMoE13B \(Activated\)Qwen \(25 models\)Qwen 1\.5 MoE \(2 models\)Qwen1\.5\-moe\-a2\.7b\-chatChatMoE2\.7B \(Activated\)Qwen1\.5\-moe\-a2\.7bBaseMoE2\.7B \(Activated\)Qwen 2 \(5 models\)Qwen2\-7bBaseDense7BQwen2\-0\.5b\-instructInstructDense0\.5BQwen2\-1\.5b\-instructInstructDense1\.5BQwen2\-7b\-instructInstructDense7BQwen2\-72b\-instructInstructDense72BQwen 2\.5 \(7 models\)Qwen2\.5\-0\.5b\-instructInstructDense0\.5BQwen2\.5\-1\.5b\-instructInstructDense1\.5BQwen2\.5\-3b\-instructInstructDense3BQwen2\.5\-7b\-instructInstructDense7BQwen2\.5\-14b\-instructInstructDense14BQwen2\.5\-32b\-instructInstructDense32BQwen2\.5\-72b\-instructInstructDense72BQwen 3 \(9 models\)Qwen3\-0\.6b\-instructInstructDense0\.6BQwen3\-1\.7b\-instructInstructDense1\.7BQwen3\-4bBaseDense4BQwen3\-4b\-instructInstructDense4BQwen3\-8bBaseDense8BQwen3\-8b\-instructInstructDense8BQwen3\-14bBaseDense14BQwen3\-14b\-instructInstructDense14BQwen3\-32b\-instructInstructDense32BQwen 3 A3B \(2 models\)Qwen3\-30b\-a3b\-instructInstructMoE3B \(Activated\)Qwen3\-30b\-a3bBaseMoE3B \(Activated\)Olmo \(9 models\)Olmo v1 \(3 models\)Olmo\-1bBaseDense1BOlmo\-7bBaseDense7BOlmo\-7b\-instructInstructDense7BOlmo v2 \(4 models\)Olmo\-2\-1b\-instructInstructDense1BOlmo\-2\-7b\-instructInstructDense7BOlmo\-2\-13b\-instructInstructDense13BOlmo\-2\-32b\-instructInstructDense32BOlmoE \(2 models\)Olmoe\-7bBaseMoE1B \(Activated\)Olmoe\-7b\-instructInstructMoE1B \(Activated\)InternLM \(2 models\)InternLM 2 \(2 models\)Internlm2\-7bBaseDense7BInternlm2\-chat\-7bChatDense7B
## Appendix EExperimental Details

### E\.1Computing Infrastructure

We conducted all our experiments on four NVIDIA Tesla V100\-DGXS GPUs, each with 32 GB of VRAM\. The software environment consisted of NVIDIA Driver version 570\.133\.07 and CUDA 12\.8\.

For all of the evaluated LLMs, we used a torch\.float16 data type, which provides a standard level of precision for inference tasks\.

### E\.2Model Details

We conduct experiments on 50 open\-source LLMs from 6 major model families\. A comprehensive list of all evaluated models is provided in Table[1](https://arxiv.org/html/2608.18539#A4.T1)\. To facilitate a controlled analysis of the factors influencing prompt sensitivity, we group these models into specific subsets for each comparison, as detailed below\.

\(1\)Instruct/Chat vs\. Base Models\.To investigate the impact of the alignment process, we form pairs of instruct/chat models and their corresponding base models\. This comparison includes models from the Llama, Mistral, Qwen, InternLM, and Olmo families\. The main LLMs used for this comparison are:

- •Llama Family: - –llama\-2\-7b\-chatvs\.llama\-2\-7b - –llama\-2\-13b\-chatvs\.llama\-2\-13b - –llama\-2\-70b\-chatvs\.llama\-2\-70b - –llama\-3\-8b\-instructvs\.llama\-3\-8b
- •Mistral Family: - –mistral\-7b\-v0\.3\-instructvs\.mistral\-7b\-v0\.3 - –mixtral\-8x7b\-instructvs\.mixtral\-8x7b
- •Qwen Family: - –qwen3\-4b\-instructvs\.qwen3\-4b - –qwen3\-8b\-instructvs\.qwen3\-8b - –qwen3\-14b\-instructvs\.qwen3\-14b - –qwen1\.5\-moe\-a2\.7b\-chatvs\.qwen1\.5\-moe\-a2\.7b - –qwen3\-30b\-a3b\-instructvs\.qwen3\-30b\-a3b
- •Olmo Family: - –olmo\-7b\-instructvs\.olmo\-7b - –olmoe\-7b\-instructvs\.olmoe\-7b
- •InternLM Family: - –internlm2\-chat\-7bvs\.internlm2\-7b

\(2\)Dense vs\. MoE Models\.To analyze the effect of architecture, we compare dense and Mixture\-of\-Experts \(MoE\) models, primarily within the same model family to control for other variables\. The main LLMs used for this comparison are:

- •Llama Family: - –Dense:llama\-2\-7b,llama\-2\-7b\-chat,llama\-2\-13b,llama\-2\-13b\-chat,llama\-2\-70b,llama\-2\-70b\-chat,llama\-3\-8b,llama\-3\-8b\-instruct\. - –MoE:llama\-moe\-v1\-3\_5b\-2\_8\-sft,llama\-moe\-v2\-3\_8b\-2\_8\-sft\.
- •Mistral Family: - –Dense:mistral\-7b\-v0\.3,mistral\-7b\-v0\.3\-instruct\. - –MoE:mixtral\-8x7b,mixtral\-8x7b\-instruct\.
- •Qwen Family: - –Dense:qwen2\-7b,qwen2\-7b\-instruct,qwen2\-72b\-instruct,qwen2\.5\-7b\-instruct,qwen2\.5\-14b\-instruct,qwen2\.5\-32b\-instruct,qwen2\.5\-72b\-instruct,qwen3\-4b,qwen3\-4b\-instruct,qwen3\-8b,qwen3\-8b\-instruct,qwen3\-14b,qwen3\-14b\-instruct,qwen3\-32b\-instruct\. - –MoE:qwen1\.5\-moe\-a2\.7b,qwen1\.5\-moe\-a2\.7b\-chat,qwen3\-30b\-a3b,qwen3\-30b\-a3b\-instruct\.
- •Olmo Family: - –Dense:olmo\-7b,olmo\-7b\-instruct,olmo\-2\-7b\-instruct,olmo\-2\-13b\-instruct,olmo\-2\-32b\-instruct\. - –MoE:olmoe\-7b,olmoe\-7b\-instruct\.

\(3\)Model Scale\.To study the impact of model scale, we analyze a series of LLMs from the same family and with the same training paradigm but with varying parameter counts\. The main LLMs used for this comparison are:

- •Llama\-2 \(Base\):7b,13b,70b\.
- •Llama\-2 \(Chat\):7b,13b,70b\.
- •Qwen2 \(Instruct\):0\.5b,1\.5b,7b,72b\.
- •Qwen2\.5 \(Instruct\):0\.5b,1\.5b,3b,7b,14b,32b,72b\.
- •Qwen3 \(Instruct\):0\.6b,1\.7b,4b,8b,14b,32b\.
- •Olmo\-2 \(Instruct\):1b,7b,13b,32b\.

### E\.3Generation Configuration of LLMs

To ensure reproducible results, we employ a greedy search strategy for all LLMs\. This is achieved by setting the “do\_sample” parameter to “False” in our generation configuration\. When “do\_sample=False”, the LLM selects the token with the highest probability as the next token in the sequence\. By adopting this greedy approach, we eliminate the randomness inherent in sampling\-based methods\. The configuration ensures that for a given input, the same LLM will generate the exact same output every time, which is a critical requirement for the replicability of our experiments\.

### E\.4How to Mask Input Words For Different LLMs

To compute interactions, we follow the approach of[12](https://arxiv.org/html/2608.18539#bib.bib24)and mask the words inN∖SN\\setminus Sby replacing them with a LLM\-specific\[MASK\]token\. Our selection of this token follows a prioritized strategy: \(1\) We preferentially use the LLM’s designated unknown \(<unk\>\) token\. \(2\) If an unknown token is not available or suitable, we use the padding \(<pad\>\) token as a fallback\. Since the specific token strings and their corresponding IDs vary across different LLMs, the exact mask token used for each LLM is detailed below:

- •Forllama\-2\-7b,llama\-2\-7b\-chat,llama\-moe\-v1\-3\.5b\-2\.8\-sft,mistral\-7b\-v0\.3,mistral\-7b\-v0\.3\-instruct,mixtral\-8x7b,mixtral\-8x7b\-instruct,internlm2\-7b,internlm2\-chat\-7b, we use the<unk\>token \(ID:0\) to mask words\.
- •Forllama\-2\-13b,llama\-2\-13b\-chat,llama\-2\-70b,llama\-2\-70b\-chat, we use the<\|pad\_token\|\>token \(ID:0\) to mask words\.
- •Forllama\-3\-8b,llama\-3\-8b\-instruct, we use the<\|pad\_token\|\>/<\|reserved\_special\_token\_250\|\>token \(ID:128255\) to mask words\.
- •Forllama\-moe\-v2\-3\.8b\-2\.8\-sft, we use the<\|pad\_token\|\>/<\|eot\_id\|\>token \(ID:128009\) to mask words\.
- •Forqwen2\-7b, we use the<\|PAD\_TOKEN\|\>token \(ID:151646\) to mask words\.
- •For a large group of Qwen models, includingqwen2\-0\.5b\-instruct,qwen2\-1\.5b\-instruct,qwen2\-7b\-instruct,qwen2\-72b\-instruct,qwen2\.5series,qwen3\-0\.6b\-instruct,qwen3\-1\.7b\-instruct,qwen3\-4bseries,qwen3\-32b\-instruct,qwen1\.5\-moeseries, andqwen3\-30b\-a3bseries, we use the<\|pad\_token\|\>/<\|endoftext\|\>token \(ID:151643\) to mask words\.
- •Forqwen3\-8b,qwen3\-8b\-instruct,qwen3\-14b,qwen3\-14b\-instruct, we use the<\|pad\_token\|\>/<\|vision\_pad\|\>token \(ID:151654\) to mask words\.
- •For the Olmo V1 series models, includingolmo\-1b,olmo\-7b,olmo\-7b\-instruct,olmoe\-7b, andolmoe\-7b\-instruct, we use the<\|padding\|\>token \(ID:1\) to mask words\.
- •For the Olmo V2 series models, includingolmo\-2\-1b\-instruct,olmo\-2\-7b\-instruct,olmo\-2\-13b\-instruct, andolmo\-2\-32b\-instruct, we use the<\|pad\_token\|\>/<\|endoftext\|\>token \(ID:100257\) to mask words\.

### E\.5Prompt Templates

![Refer to caption](https://arxiv.org/html/2608.18539v1/prompt_templates.png)Figure 15:Different prompt templates\. Red parts show the difference between the current prompt template with the first prompt template,i\.e\., Prompt Template 1\.To systematically evaluate the prompt sensitivity of LLMs, we designed a set of five distinct prompt templates\. As illustrated in Figure[15](https://arxiv.org/html/2608.18539#A5.F15), these templates are derived from a base prompt template \(i\.e\., Prompt Template 1\) through a series of subtle, semantically irrelevant modifications\. These variations include changes in letter case,e\.g\., “Answers” vs\. “ANSWERS” and alterations to separators,e\.g\., “:” vs\. “::” or the format of option markers,e\.g\., “A\.” vs\. “A\)”\. Crucially, these changes only affect the superficial formatting while preserving the core semantic meaning of the prompt template\.

In our experimental procedure, for a given input, which consists of a question and options, we apply each of the five prompt templates to generate five prompts\. For every unique pair of these five prompts, we then calculate the prompt sensitivity by quantifying the change in the interactions among the input variables \(i\.e\., words within the question and options\)\. This procedure allows us to precisely measure how much the LLM’s interaction patterns of the core input are perturbed by superficial changes in the prompt template, thus evaluating the prompt sensitivity of LLMs\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/fsl_templates.png)Figure 16:Prompt templates of few\-shot learning\.
### E\.6Few\-shot Learning Templates

For this experiment, we selected the pair of prompt templates that exhibited the highest average prompt sensitivity in the 0\-shot setting, aiming to test if few\-shot learning could help the most severe situation\. To investigate whether few\-shot learning can mitigate high prompt sensitivity, we conducted a follow\-up experiment\. We selectedPrompt Template 1andPrompt Template 4from Figure[15](https://arxiv.org/html/2608.18539#A5.F15)for this analysis, as this pair exhibited the highest average prompt sensitivity in our 0\-shot setting\. This allowed us to test the efficacy of few\-shot learning in the most challenging scenario\.

Based on these two base templates, we constructed few\-shot learning prompts with one, two, and three in\-context examples \(i\.e\., 1\-shot, 2\-shot, and 3\-shot learning\), as illustrated in Figure[16](https://arxiv.org/html/2608.18539#A5.F16)\. The examples were formulated using certain questions and their corresponding answers, randomly selected from a set of datasets that are not included in the test set\. The structure of each example is related to its corresponding prompt template\. For instance, the first example \(Example 1\) is formatted differently for each template:

- •Example 1 For Prompt Template 1: Question:Whichtypeofprecipitationconsistsoffrozenraindrops? Answers: A\.sleet B\.hail C\.snow D\.fog Answer:A
- •Example 1 For Prompt Template 4 \(Note the different format\): Question:Whichtypeofprecipitationconsistsoffrozenraindrops? ANSWERS:: A\.sleet B\.hail C\.snow D\.fog ANSWER::A

The other two examples \(Example 2andExample 3\) are presented below:

- •Example 2 For Prompt Template 1: Question:Decayedprehistoricplantshavehelpedintheformationof Answers: A\.coal,shale,andquartz\. B\.coal,oil,andgas\. C\.shale,quartz,andcoal\. D\.oil,shale,andgranite\. Answer:B
- •Example 2 For Prompt Template 4: Question:Decayedprehistoricplantshavehelpedintheformationof ANSWERS:: A\.coal,shale,andquartz\. B\.coal,oil,andgas\. C\.shale,quartz,andcoal\. D\.oil,shale,andgranite\. ANSWER::B
- •Example 3 For Prompt Template 1: Question:Whichdescribesamaterialthatisnotafood? Answers: A\.Itstoresenergybutnotnutrients\. B\.Itdoesnotstoreenergyornutrients\. C\.Itstoresenergyandnutrients\. D\.Itdoesnotstoreenergybutstoresnutrients\. Answer:B
- •Example 3 For Prompt Template 4: Question:Whichdescribesamaterialthatisnotafood? ANSWERS:: A\.Itstoresenergybutnotnutrients\. B\.Itdoesnotstoreenergyornutrients\. C\.Itstoresenergyandnutrients\. D\.Itdoesnotstoreenergybutstoresnutrients\. ANSWER::B

## Appendix FMore Experimental Results

### F\.1More Results on the Verification of the Sparsity of Interactions

Here are more results on the verification of the sparsity of interactions\. As illustrated in Figure[17](https://arxiv.org/html/2608.18539#A6.F17), the results verify that only a small set of interactions have salient effects, while most of the interactions have negligible effects and can be considered as noise patterns\.

Figure 17:Verifying the sparsity of interactions\. We show absolute values of normalized interactions in a descending order\. LLMs all encode a small number of salient interactions, while most of the interaction effects are negligible\.
### F\.2More Results on the Verification of the Sparsity of Interactions

Here are more results on the verification of quality of universal matching\. Figure[18](https://arxiv.org/html/2608.18539#A6.F18)compares the LLM’s true outputv⁡\(𝒙T\)v\(\\bm\{x\}\_\{T\}\)for all masked inputs against the logical model using only the most salient interactions\. Even when using just the top 3% or top 5% of all interactions, the matching error is minimal\. This empirically demonstrates that the LLM’s output can be faithfully approximated by a small, sparse set of salient interactions\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/mimic.png)Figure 18:Verifying the quality of universal matching for any2n2^\{n\}masked inputs\. The red line plots outputs of the LLM in an ascending order\.
### F\.3Detailed Case Study

Figure[19](https://arxiv.org/html/2608.18539#A6.F19)is the detailed case study of how to use our interaction\-based analytical tool\. It offers preliminary evidence that semantically irrelevant alterations to the prompt template can lead to significant changes in the salient interaction patterns, even when the input and output remains unchanged\. This reveals the existence of unstable interactions, which we propose as the underlying cause of prompt sensitivity\.

Figure 19:A case study of interaction\-level analysis revealing latent instability\. The same input𝒙\\bm\{x\}is formatted with two semantically identical templates,TTandT^\\hat\{T\}, differing only in letter case \(e\.g\.,“Answer” vs\. “ANSWER”\)\. Although the LLM generates the same correct output \(“D”\) in both cases, the composition of the interaction\-based logical modelϕ⁡\(𝒙\)\\phi\(\\bm\{x\}\)reveals significant internal divergence\. Many interaction effects are highly unstable, changing in either sign or magnitude\. This highlights a critical risk of prompt sensitivity that is invisible to output\-level
### F\.4More Results on the Prompt Sensitivity of Different Orders

Here are more results on the prompt sensitivity of different orders on the ARC dataset\. As illustrated in Figure[20](https://arxiv.org/html/2608.18539#A6.F20), it shows that the prompt sensitivity of low\-order interactions is the lowest, followed by mid\-order, while high\-order interactions exhibit the highest prompt sensitivity\. This indicates that low\-order interactions encoded by LLMs are highly stable when faced with subtle changes to prompt templates,i\.e\., simple interaction patterns are more robust\. Conversely, the high sensitivity of high\-order interactions reveals that the LLMs’ internal representation of complex patterns is highly unstable\.

Figure 20:A comparison of the prompt sensitivity of three order types\. Results show that low\-order interactions are the least sensitive, while high\-order interactions are the most sensitive\.
### F\.5More Results on Relative Change in the Prompt Sensitivity of Low\-, Mid\-, and High\-Order Interactions for Different Factors\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_0.1.png)Figure 21:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.
### F\.6More Results on the Prompt Sensitivity of Different Order Types across Different Model Scales\.

Figure 22:A comparison of prompt sensitivity of different order types across different model scales\.
### F\.7More Results on the Prompt Sensitivity of Different Orders for Each Individual LLM when Applying Few\-Shot Learning

Figure 23:A comparison of prompt sensitivity of low\-, mid\-, and high\-order interactions between 0\-shot learning and few\-shot learning\. Prompt sensitivity at all three order levels shows an clear drop when applying few\-shot learning\.
### F\.8Hyperparameter Experiments of the thresholdτ\\tau

To rigorously evaluate the robustness of our Interaction\-based Prompt Sensitivity \(IPS\) metric, we conducted a hyperparameter sweep on the thresholdτ\\tau\.

#### F\.8\.1Detailed Model Rankings under Varying Thresholds

We aggregated the ranking and scoring consistency across all 16 thresholds using five metrics\. As shown in Table[2](https://arxiv.org/html/2608.18539#A6.T2), the high correlation coefficients and low error rates demonstrate that the IPS metric is highly robust to the choice ofτ\\tau\.

Table 2:Summary of consistency metrics across 16 differentτ\\tauthresholds \(0\.05–0\.20\)\. The high values in correlation metrics and low RMSE indicate that the relative ranking of model sensitivity remains stable regardless of the specific threshold used\.To provide a granular view of robustness, Table[3](https://arxiv.org/html/2608.18539#A6.T3)details the IPS scores across 10 distinct thresholds ranging fromτ=0\.05\\tau=0\.05toτ=0\.20\\tau=0\.20\. Models are sorted based on their stability at the baseline thresholdτ=0\.05\\tau=0\.05\. The data reveals that while absolute scores fluctuate, the relative ranking of model stability remains highly consistent\.

Table 3:Detailed IPS scores for 50 LLMs across 10 different thresholds\. The consistency in color gradients \(implied by values\) across rows confirms the robustness of the metric\.IPS Score \(↓\\downarrow\) at Thresholdτ\\tauModel Name0\.050\.060\.070\.080\.090\.100\.120\.150\.180\.20Qwen2\.5\-72B\-Instruct1\.3281\.3121\.2971\.2861\.2761\.2681\.2551\.2401\.2281\.222Qwen2\-72B\-Instruct1\.3621\.3461\.3331\.3221\.3121\.3041\.2901\.2781\.2711\.265Qwen2\-7B\-Instruct1\.4111\.4001\.3911\.3831\.3761\.3681\.3571\.3411\.3271\.318Qwen3\-32B\-Instruct1\.4241\.4191\.4151\.4111\.4081\.4061\.4031\.4011\.3961\.395Qwen3\-14B\-Instruct1\.4431\.4381\.4341\.4291\.4271\.4251\.4191\.4151\.4121\.407Llama\-2\-70B\-Chat1\.4501\.4461\.4411\.4381\.4361\.4331\.4281\.4201\.4081\.398Qwen2\.5\-32B\-Instruct1\.4631\.4541\.4461\.4401\.4341\.4301\.4231\.4151\.4111\.409Llama\-3\-8B\-Instruct1\.4861\.4831\.4801\.4771\.4751\.4731\.4671\.4601\.4471\.436Qwen2\.5\-14B\-Instruct1\.4901\.4811\.4731\.4651\.4591\.4541\.4461\.4361\.4301\.425Qwen3\-4B\-Instruct1\.4961\.4931\.4901\.4881\.4851\.4841\.4801\.4731\.4701\.464Qwen2\-1\.5B\-Instruct1\.4971\.4941\.4921\.4871\.4831\.4771\.4651\.4471\.4291\.416Llama\-2\-70B1\.5051\.5041\.5041\.5051\.5051\.5071\.5091\.5161\.5231\.527Qwen3\-8B\-Instruct1\.5151\.5111\.5081\.5051\.5021\.4991\.4941\.4911\.4911\.490Qwen3\-8B1\.5531\.5541\.5541\.5541\.5541\.5551\.5571\.5581\.5571\.555Qwen3\-4B1\.5551\.5621\.5661\.5701\.5731\.5751\.5781\.5791\.5741\.572Llama\-2\-13B\-Chat1\.5591\.5551\.5511\.5471\.5441\.5411\.5351\.5281\.5201\.514Qwen3\-14B1\.5591\.5581\.5571\.5561\.5561\.5551\.5531\.5521\.5491\.548Qwen2\-7B1\.5701\.5741\.5761\.5771\.5781\.5781\.5791\.5801\.5741\.569Qwen3\-30B\-A3B\-Instruct1\.5751\.5751\.5761\.5771\.5791\.5801\.5821\.5831\.5831\.582Qwen2\.5\-7B\-Instruct1\.5801\.5751\.5721\.5691\.5671\.5651\.5641\.5621\.5631\.563Llama\-2\-13B1\.5921\.6021\.6101\.6171\.6231\.6281\.6361\.6451\.6531\.655Olmo\-1B1\.6011\.6101\.6171\.6231\.6281\.6321\.6391\.6471\.6521\.655Olmo\-7B\-Instruct1\.6041\.5991\.5951\.5921\.5891\.5871\.5831\.5791\.5781\.576Llama\-3\-8B1\.6181\.6281\.6371\.6441\.6501\.6551\.6641\.6741\.6791\.682Qwen2\.5\-3B\-Instruct1\.6221\.6181\.6151\.6111\.6081\.6061\.6011\.5951\.5901\.585Olmo\-2\-13B\-Instruct1\.6251\.6271\.6281\.6291\.6301\.6311\.6321\.6331\.6321\.631Olmo\-2\-7B\-Instruct1\.6261\.6271\.6271\.6281\.6291\.6291\.6291\.6271\.6281\.626Olmo\-2\-32B\-Instruct1\.6261\.6261\.6251\.6241\.6231\.6231\.6221\.6201\.6191\.618Qwen3\-30B\-A3B1\.6291\.6311\.6341\.6371\.6391\.6421\.6461\.6511\.6551\.656Mistral\-7B\-v0\.3\-Instruct1\.6301\.6291\.6291\.6291\.6291\.6311\.6331\.6381\.6411\.644InternLM2\-Chat\-7B1\.6341\.6391\.6411\.6441\.6471\.6501\.6561\.6601\.6641\.667Qwen2\-0\.5B\-Instruct1\.6391\.6501\.6581\.6631\.6661\.6691\.6691\.6651\.6581\.651Mixtral\-8x7B\-Instruct1\.6411\.6381\.6361\.6331\.6321\.6311\.6291\.6281\.6291\.630Qwen1\.5\-MoE\-A2\.7B1\.6451\.6581\.6681\.6751\.6821\.6871\.6961\.7051\.7121\.717Qwen1\.5\-MoE\-A2\.7B\-Chat1\.6481\.6541\.6571\.6601\.6631\.6651\.6671\.6681\.6641\.660Qwen3\-1\.7B\-Instruct1\.6481\.6451\.6421\.6411\.6401\.6391\.6391\.6371\.6381\.637Qwen2\.5\-0\.5B\-Instruct1\.6521\.6631\.6711\.6771\.6811\.6841\.6881\.6891\.6871\.685Qwen2\.5\-1\.5B\-Instruct1\.6581\.6631\.6671\.6711\.6741\.6761\.6791\.6841\.6861\.689Llama\-MoE\-v2\-3\_8B\-2\_8\-SFT1\.6651\.6691\.6731\.6761\.6781\.6811\.6851\.6911\.6961\.698Llama\-2\-7B\-Chat1\.6701\.6771\.6831\.6871\.6901\.6911\.6921\.6931\.6911\.689Olmo\-7B1\.6711\.6791\.6851\.6901\.6951\.6991\.7061\.7141\.7181\.721Mixtral\-8x7B1\.6771\.6821\.6861\.6891\.6921\.6951\.7011\.7091\.7151\.720Olmoe\-7B1\.6801\.6891\.6961\.7021\.7071\.7111\.7201\.7331\.7421\.746Qwen3\-0\.6B\-Instruct1\.6821\.6781\.6751\.6731\.6711\.6681\.6651\.6591\.6541\.653Llama\-2\-7B1\.6821\.6921\.6991\.7051\.7111\.7161\.7241\.7331\.7401\.745Olmo\-2\-1B\-Instruct1\.6981\.7101\.7191\.7271\.7341\.7391\.7471\.7561\.7601\.760InternLM2\-7B1\.7061\.7111\.7161\.7201\.7241\.7281\.7341\.7431\.7511\.758Llama\-MoE\-v1\-3\_5B\-2\_8\-SFT1\.7151\.7201\.7241\.7281\.7321\.7351\.7431\.7531\.7611\.765Olmoe\-7B\-Instruct1\.7161\.7171\.7191\.7211\.7221\.7231\.7241\.7261\.7301\.732Mistral\-7B\-v0\.31\.7201\.7291\.7371\.7431\.7481\.7521\.7601\.7681\.7751\.779
#### F\.8\.2Verifying the generalizability of the methods and conclusions on different thresholdτ\\tau\.

In Sections[4\.2](https://arxiv.org/html/2608.18539#S4.SS2)and[4\.3](https://arxiv.org/html/2608.18539#S4.SS3), we set the thresholdτ\\tauto 0\.1 to distinguish salient interactions from noise\. This threshold directly influences the proportion of interactions classified as salient interactions\. A higherτ\\tauvalue usually generates a smaller set of salient interactions with more significant effects\.[27](https://arxiv.org/html/2608.18539#bib.bib17)conducted experiments which show that conclusions are not sensitive to the choice ofτ\\tau\. Our choice ofτ\\tauis guided by the empirical sparsity of interactions\. Figure[2](https://arxiv.org/html/2608.18539#S3.F2)\(a\) shows a sharp “elbow” in the distribution of interaction effects, clearly separating a small set of high\-magnitude salient interactions from a long tail of near\-zero noise interactions\. A thresholdτ\\tauchosen from the range of 0\.05 to 0\.15 effectively captures this salient set, satisfying the sparsity assumption without being overly restrictive\. To ensure the robustness of our findings, we conducted hyperparameter experiments withτ=0\.05\\tau=0\.05andτ=0\.15\\tau=0\.15\. Our main conclusions remain consistent across different threshold values\.

Here are the results forτ=0\.15\\tau=0\.15\.

Figure 24:A comparison of the prompt sensitivity between instruct/chat models and base models\. Results show that instruct/chat models are less sensitive than corresponding base models\.
Figure 25:A comparison of prompt sensitivity across different model scales\. As the model scale increases, the prompt sensitivity within a model series systematically decreases\.

Figure 26:A Comparison of prompt sensitivity between MoE models and dense models\. Generally, MoE models tend to be more sensitive than dense models in the same model family\.
Figure 27:A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\. The drop in prompt sensitivity is substantial from 0\-shot to 1\-shot\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_0.15.png)Figure 28:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 29:A comparison of the prompt sensitivity of three order types\. Results show that low\-order interactions are the least sensitive, while high\-order interactions are the most sensitive\.Figure 30:A comparison of prompt sensitivity at the order\-level across different model scales\.Here are the results forτ=0\.05\\tau=0\.05\.

Figure 31:A comparison of the prompt sensitivity between instruct/chat models and base models\. Results show that instruct/chat models are less sensitive than corresponding base models\.
Figure 32:A comparison of prompt sensitivity across different model scales\. As the model scale increases, the prompt sensitivity within a model series systematically decreases\.

Figure 33:A Comparison of prompt sensitivity between MoE models and dense models\. Generally, MoE models tend to be more sensitive than dense models in the same model family\.
Figure 34:A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\. The drop in prompt sensitivity is substantial from 0\-shot to 1\-shot\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_0.05.png)Figure 35:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 36:A comparison of the prompt sensitivity of three order types\. Results show that low\-order interactions are the least sensitive, while high\-order interactions are the most sensitive\.Figure 37:A comparison of prompt sensitivity at the order\-level across different model scales\.

### F\.9Results on MMLU Dataset

Here are the results on the MMLU dataset andτ\\tauis set to 0\.1, we can observe the same conclusions on this dataset\.

Figure 38:A comparison of the prompt sensitivity between instruct/chat models and base models\. Results show that instruct/chat models are less sensitive than corresponding base models\.
Figure 39:A comparison of prompt sensitivity across different model scales\. As the model scale increases, the prompt sensitivity within a model series systematically decreases\.

Figure 40:A Comparison of prompt sensitivity between MoE models and dense models\. Generally, MoE models tend to be more sensitive than dense models in the same model family\.
Figure 41:A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\. The drop in prompt sensitivity is substantial from 0\-shot to 1\-shot\.

![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_mmlu.png)Figure 42:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 43:A comparison of the prompt sensitivity of three order types\. Results show that low\-order interactions are the least sensitive, while high\-order interactions are the most sensitive\.Figure 44:A comparison of prompt sensitivity at the order\-level across different model scales\.

## Appendix GPrompt Sensitivity of altering tokens vs\. adding tokens

Disaggregating prompt sensitivity by the type of perturbation offers deeper insights into model behavior\. Following this direction, we conduct a fine\-grained decomposition of our experimental results, comparing prompt sensitivity of two distinct categories to prompt alterations:

1. 1\.Altering tokens:This involves modifying the capitalization of words, such as from"Answer"to"ANSWER"\.
2. 2\.Adding tokens:This involves adding symbolic components, for instance, changing a colon from":"to"::"\.

Table 4:IPS Scores of Dense Models on Different Perturbation Types\. Higher scores indicate greater sensitivity\. The more sensitive perturbation type for each model is highlighted in bold\.Table 5:IPS Scores of MoE Models on Different Perturbation Types\. The more sensitive perturbation type for each model is highlighted in bold\.Our results highlight a clear architectural divide:\(1\) Dense models are more sensitive to adding tokens\.As shown in Table[4](https://arxiv.org/html/2608.18539#A7.T4), 9 out of the 10 analyzed dense models exhibit greater sensitivity to the ”adding tokens” category\. This suggests a strong, consistent trend where changes to the template’s structure have a more pronounced impact on the internal interactions of dense architectures\.\(2\) MoE models are more sensitive to altering tokens\.In stark contrast, Table[5](https://arxiv.org/html/2608.18539#A7.T5)shows that 5 out of the 6 MoE models are more sensitive to ”altering tokens”\. This consistent pattern suggests that MoE architectures are more susceptible to variations in the change of word capitalization\.

## Appendix HComparison between IPS and Other Metrics

To demonstrate the unique value of our interaction\-based approach, we compare the Interaction\-based Prompt Sensitivity \(IPS\) against standard coarse\-grained metrics derived from internal representations\. Specifically, we measure theCosine SimilarityandL2L\_\{2\}Distanceof the final\-layer hidden states under prompt perturbations\. While these metrics are commonly used to assess representation robustness, our analysis reveals that they fail to capture the nuanced mechanisms of prompt sensitivity in LLMs\.

### H\.1Empirical Inconsistency of Representation\-based Metrics

We re\-evaluateFactor 1 \(Base vs\. Instruct/Chat models\)using these representation\-based metrics\. The results, summarized in Table[6](https://arxiv.org/html/2608.18539#A8.T6), demonstrate a significant lack of consistency compared to the robust trends observed via IPS\.

Table 6:Comparison of stability metrics \(Cosine Similarity andL2L\_\{2\}Distance\) for Base vs\. Instruct/Chat models\. Unlike IPS, these metrics fail to show a consistent trend regarding the impact of Supervised Fine\-Tuning\.Base ModelInstruct/Chat ModelModel FamilyCosine\(↑\\uparrow\)L2L\_\{2\}Dist\.\(↓\\downarrow\)Cosine\(↑\\uparrow\)L2L\_\{2\}Dist\.\(↓\\downarrow\)Llama\-2\-7B0\.7700\.77075\.120\.78166\.5166\.51Llama\-2\-13B0\.94217\.350\.9270\.92719\.6919\.69Llama\-2\-70B0\.91133\.650\.7950\.79543\.1443\.14Llama\-3\-8B0\.8560\.85679\.7779\.770\.88073\.56Mistral\-7B\-V0\.30\.8110\.811207\.89207\.890\.843154\.43Mixtral\-8x7B0\.965138\.71138\.710\.8750\.875136\.03InternLM2\-7B0\.8980\.898142\.51142\.510\.98775\.83Olmo\-7B0\.79336\.880\.6610\.66148\.3348\.33Olmoe\-7B0\.67256\.670\.5100\.51085\.7385\.73Qwen2\-7B0\.8640\.864138\.73138\.730\.888136\.49Qwen3\-4B0\.91460\.130\.7790\.77973\.8273\.82Qwen3\-8B0\.99533\.950\.9620\.96241\.4241\.42Qwen3\-14B0\.94767\.8567\.850\.9320\.93251\.97Qwen1\.5\-MoE\-A2\.7B0\.7650\.765160\.31160\.310\.91186\.98Qwen3\-30B\-A3B0\.8640\.86481\.1381\.130\.93239\.04As shown in Table[6](https://arxiv.org/html/2608.18539#A8.T6), neither Cosine Similarity norL2L\_\{2\}Distance provides a reliable proxy for prompt sensitivity:

- •Contradictions between metrics:For models likeMixtral\-8x7B, the two metrics contradict each other—Cosine Similarity suggests the Base model is more stable, whileL2L\_\{2\}Distance favors the Instruct model\.
- •Inconsistency with established trends:While our IPS analysis \(and general consensus\) identifies Instruct/Chat models as more robust to prompt variations, representation metrics frequently suggest the opposite\. For instance, in theLlama\-2\-13BandQwen3\-8Bpairs, the Base models exhibit higher cosine similarity and lowerL2L\_\{2\}distance than their Instruct counterparts\.
- •Random fluctuations:There is no discernible pattern across model families\. ForLlama\-2\-7B, the Chat version appears more stable via Cosine Similarity but less stable viaL2L\_\{2\}Distance\.

These contradictions indicate that global measures of hidden state changes are too coarse to serve as accurate indicators of the model’s functional sensitivity\.

### H\.2Superiority of the Interaction\-based Framework

The empirical limitations of representation\-based metrics highlight the theoretical advantages of our proposed framework\. The superiority of IPS stems from two fundamental differences:

##### 1\. Explanability \(“Why” vs\. “What”\):

Hidden state similarity merely measureswhathas changed—the magnitude or direction of the aggregate internal representation vector\. It treats the model as a black box regarding the reasoning process\. In contrast, our interaction framework explainswhythe output fluctuates\. By decomposing predictions into interactions, we can pinpoint specific combinations of input tokens \(inference patterns\) that become unstable\. This fine\-grained insight allows us to distinguish between benign representation shifts and those that disrupt the model’s logical coherence\.

##### 2\. Faithfulness to the Output:

Representation metrics lack a direct mathematical link to the final prediction\. A small shift in Euclidean distance can sometimes lead to a flipped prediction, while a large shift might not\. Conversely, our method is grounded in theUniversal Matching Property\(Theorem 1\)\. This theorem guarantees that the sum of all interactions perfectly reconstructs the LLM’s output score\. Consequently, IPS provides a faithful evaluation of the decision\-making logic, ensuring that the measured sensitivity directly reflects the instability in the model’s actual predictive mechanism\.

## Appendix IDetailed Discussion on the Impact of Model Architecture

We notice that in Figure[7](https://arxiv.org/html/2608.18539#S4.F7), the behavior of Mistral family is different from the other family: the dense models are more sensitive than the MoE models at low\-order and mid\-order level\. We attribute this to the number of activated parameters, extending our finding from Factor 2 that larger models are less sensitive\. While most MoE models \(e\.g\., in Llama and Qwen families\) are more sensitive due to fewer activated parameters, the Mistral case is reversed: Mixtral\-8x7B activates more parameters than its dense counterpart Mistral\-7B\-V0\.3 \(13B vs\. 7B\), resulting in lower prompt sensitivity\.

To more rigorously isolate the influence of model architecture on the prompt sensitivity from model size or other factors, we conduct a controlled variable analysis\. We select specific pairs from the Qwen and OLMo families that share the most similar model scales \(i\.e\., activated parameters\) and analogous training paradigms \(i\.e\., base vs\. base, instruct/chat vs\. instruct/chat\)\. This targeted comparison enables us to minimize confounding factors and focus directly on the architectural impact\.

Results in Table[7](https://arxiv.org/html/2608.18539#A9.T7)consistently show that MoE models exhibit higher prompt sensitivity than dense models\. This suggests that the increased prompt sensitivity of MoE architectures is not merely a consequence of smaller number of activated parameters\. Instead, it further strengthens our conclusion that the MoE architecture inherently increases the prompt sensitivity of LLMs\.

Table 7:Controlled variable analysis of prompt sensitivity between MoE and dense models\.
## Appendix JSolutions for reducing the computational cost of the method

The limitation of our current study lies in the computational cost of the interaction framework\. The method’s complexity scales exponentially with the number of input variablesnn, as it requires evaluating2n2^\{n\}masked inputs\. However, applying this method to very long text inputs would demand a high computational load\.

Future work can address this scalability challenge through several promising avenues\. These strategies aim to reduce the effective number of input variables without fundamentally changing the faithfulness of the analysis:

\(1\)Selective Input Variable Analysis\.One approach is to analyze only a subset of informative input variables \(i\.e\., words\) while treating uninformative ones \(e\.g\., stop words\) as fixed background context\. Previous research has demonstrated that this selection does not significantly impair the faithfulness of the interaction framework\([11](https://arxiv.org/html/2608.18539#bib.bib20)\)\.

\(2\)Phrase\-level Aggregation\.Instead of analyzing individual words, we can operate at a coarser perspective by merging related words into combined phrasal units\. This reduces the total number of input variables while preserving key semantic meaning\.

For methods \(1\) and \(2\), we have put them into practice\. In our experiments on long\-form open\-ended questions, we apply these two methods to effectively control the number of input variables\. The specific selection strategies are detailed in Appendix[K](https://arxiv.org/html/2608.18539#A11)and Appendix[L](https://arxiv.org/html/2608.18539#A12)\. It demonstrates that these techniques can substantially reduce computational complexity without affecting the key conclusions\.

\(3\)Approximation Methods\.In addition to selection strategies, techniques specifically designed for efficiently computing sparse interactions to bypass exhaustiveO⁡\(2n\)O\(2^\{n\}\)evaluations\([26](https://arxiv.org/html/2608.18539#bib.bib57);[5](https://arxiv.org/html/2608.18539#bib.bib58)\), represent a clear path for reducing computational cost\.

These strategies represent promising directions for extending the powerful capabilities of interaction\-based analysis to a wider range of long\-text NLP tasks\.

## Appendix KDetails of the Experiments on Open\-Ended Generation Tasks

### K\.1Experimental Setup

This section illustrates the detailed setup of open\-ended question\-answering tasks, which more closely resemble real\-world user scenarios\. We employed theDatabricks Dolly\-15kdataset, an open\-source collection of instruction\-following records\. This dataset spans multiple behavioral categories as defined in the InstructGPT paper\([33](https://arxiv.org/html/2608.18539#bib.bib51)\), including brainstorming, classification, closed QA, open QA, and summarization, providing a diverse and realistic dataset for evaluating model robustness\. We test the results mainly on the Llama\-2 family\.

### K\.2Selection of Input Variables

Given that the text length in open\-ended instructions far exceeds that of MCQ tasks, a direct analysis of all words would lead to exponential computational costs\. To address this challenge, we applied the optimization strategies discussed in our limitations section:\(1\) Selective Input Variable Analysisand\(2\) Phrase\-level Aggregation\.

Our approach is guided by a systematic procedure to choose a fixed number of key input variables from the full input\. Specifically, for each input sentence, we select meaningful words or phrases to construct the set of input variablesNN\. A word is considered “meaningful” if it is not an NLTK\([4](https://arxiv.org/html/2608.18539#bib.bib53)\)stop word or a punctuation mark\. The remaining parts of the text, such as generic instruction templates \(e\.g\., ”Below is an instruction…”\) and stop words, are treated as fixed background context\. During the interaction analysis, only the variables within the setNNare masked\.

For a concrete example, consider the following prompt:

> Below is an instruction that describes a task\. Write a response that appropriately completes the request\. \#\#\# Instruction: Identify from the following list characters from The X\-Files who are bald or balding: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg \#\#\# Response:

Selection Process for Input Variables:

1. 1\.Method \(1\) Application:We designate the generic instruction template and functional stop words \(e\.g\., ”from”, ”the”, ”who”, ”are”\) as background context, excluding them from the input variable set\.
2. 2\.Method \(2\) Application:We aggregate words forming core semantic concepts into single phrasal units, such as the key entity"Walter Skinner"and the critical condition"bald or balding"\.

Final Input Variables:

```
[
  "Identify", "following list", "characters","X-Files",
  "bald or balding","Walter Skinner", "John Fitzgerald Byers",
  "Dana Scully","Melvin Frohike", "Darius Michaud",
  "Peter Watts","Conrad Strughold", "Queequeg"
]
```

### K\.3Experimental Results

By applying the aforementioned input variable selection strategies \(Methods 1 and 2\) in our experiments on the Dolly dataset, we successfully managed the analytical complexity for each long\-text input, leading to a substantial reduction in computational cost\.

Crucially, the experimental outcomes derived from open\-ended questions and the optimized setup remainedhighly consistentwith the main conclusions drawn from our MCQ\-based experiments\.

Here are the results andτ\\tauis set to 0\.1, we can observe the same conclusions in this experiment\.

Figure 45:A comparison of the prompt sensitivity between instruct/chat models and base models\. Results show that instruct/chat models are less sensitive than corresponding base models\.
Figure 46:A comparison of prompt sensitivity across different model scales\. As the model scale increases, the prompt sensitivity within a model series systematically decreases\.

Figure 47:A Comparison of prompt sensitivity between MoE models and dense models\. Generally, MoE models tend to be more sensitive than dense models in the same model family\.
Figure 48:A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\. The drop in prompt sensitivity is substantial from 0\-shot to 1\-shot\.

Figure 49:A comparison of the prompt sensitivity of three order types\. Results show that low\-order interactions are the least sensitive, while high\-order interactions are the most sensitive\.![Refer to caption](https://arxiv.org/html/2608.18539v1/order_analysis_dolly1.png)Figure 50:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 51:A comparison of prompt sensitivity at the order\-level across different model scales\.

## Appendix LDetails of the Experiments Beyond Prompt Template Modifications

### L\.1Experimental Setup

To assess the generalizability of our findings beyond superficial template modifications, we conducted an additional experiment on theDolly\-15kdataset\. This setup introduces more realistic and complex prompt perturbations that better reflect authentic user interactions, specificallysemantic paraphrasingandinstruction reordering\. We employed these techniques to test the robustness of our conclusions under more challenging conditions\. An illustrative example of a semantic paraphrase used in our experiment is shown below:

An illustrative example of the full prompt structure and the applied perturbations is shown below\.

> Original Prompt: ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Identify from the following list characters from The X-Files who are bald or balding: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: ``` Perturbation 1: Semantic Paraphrase: ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Select from the list below figures from The X-Files that are losing their hair or bald: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: ``` Perturbation 2: Instruction Reordering: ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: From The X-Files, identify characters who are bald or balding from the following list: Walter Skinner, John Fitzgerald Byers, Dana Scully, Melvin Frohike, Darius Michaud, Peter Watts, Conrad Strughold, Queequeg ### Response: ```

### L\.2Selection of Input Variables

For this experiment, we employed the same input variable selection strategy as detailed in Appendix[K\.2](https://arxiv.org/html/2608.18539#A11.SS2)\. This approach combines Selective Input Variable Analysis and Phrase\-level Aggregation to manage the computational complexity associated with longer, open\-ended instructions while preserving the core semantic elements for interaction analysis\.

### L\.3Experimental Results

The experimental outcomes from this more challenging setup verify the main conclusions of our paper\. Despite the increased complexity of the perturbations, we consistently observed that the four identified factors \(supervised fine\-tuning, increased model scale, dense architectures, and few\-shot learning\) reduce prompt sensitivity, primarily by stabilizing low\-order interactions\. This replication demonstrates that our conclusions are not confined to simple template variations but hold true in scenarios that more closely mirror authentic user interactions, thereby strengthening the generalizability of our work\.

Here are the results andτ\\tauis set to 0\.1, we can observe the same conclusions in this experiment\.

Figure 52:A comparison of the prompt sensitivity between instruct/chat models and base models\. Results show that instruct/chat models are less sensitive than corresponding base models\.
Figure 53:A comparison of prompt sensitivity across different model scales\. As the model scale increases, the prompt sensitivity within a model series systematically decreases\.

Figure 54:A Comparison of prompt sensitivity between MoE models and dense models\. Generally, MoE models tend to be more sensitive than dense models in the same model family\.
Figure 55:A comparison of prompt sensitivity between 0\-shot learning and few\-shot learning\. The drop in prompt sensitivity is substantial from 0\-shot to 1\-shot\.

Figure 56:A comparison of the prompt sensitivity of three order types\. Results show that low\-order interactions are the least sensitive, while high\-order interactions are the most sensitive\.![Refer to caption](https://arxiv.org/html/2608.18539v1/x1.png)Figure 57:Comparing the relative change in the prompt sensitivity of low\-, mid\-, and high\-order interactions for different factors\.Figure 58:A comparison of prompt sensitivity at the order\-level across different model scales\.