LLM Explainability with Counterfactual Chains and Causal Graphs
Summary
This paper proposes a four-phase method for constructing causal graphs that model LLM inference processes, using counterfactual augmentation to enable stable causal discovery and provide transparent, concept-level explainability.
View Cached Full Text
Cached at: 06/08/26, 07:14 AM
Paper page - LLM Explainability with Counterfactual Chains and Causal Graphs
Source: https://huggingface.co/papers/2606.05972
Abstract
Causal graphs are used to model large language model inference processes, enabling transparent visualization of how models perceive and organize high-level concepts for predictions through a four-phase method involving concept discovery, mapping, and MCMC-inspired counterfactual augmentation.
Causal graphsprovide a high-level language for making mechanisms transparent. Recent work usesLarge Language Models(LLMs) to recovercausal graphsof external-world processes. Instead, in this paper, we usecausal graphsto model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce anMCMC-inspiredcounterfactual augmentationprocedure that expands the sparse observational data through chains of counterfactuals. This enables stablecausal discoverywith σ-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs forpredictive fidelityandstructural stability, and theMCMC-inspiredaugmentation for convergence and downstream utility. Our results show that the discoveredcausal graphscapture meaningful dependencies consistent with LLMs’ reasoning. Together, this paper provides a foundation forconcept-level explainabilityof LLMs.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.05972
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.05972 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.05972 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.05972 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Causal-Audit: Explicit and Auditable Graph-based Reasoning via Target-Aware Causal Chain Construction
Proposes Causal-Audit, a framework for explicit and auditable causal reasoning in LLMs using target-aware causal graph construction and path-level evidence aggregation, outperforming existing methods on benchmarks.
Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts
This paper proposes a hybrid framework combining LLM summarization, a linguistically grounded Expert Index, and STAC classification to generate causal graphs from narrative texts, outperforming GPT-4o and Claude 3.5 on benchmark stories.
LLMs are not the black box you were promised
An article summarizing Anthropic's 2025 paper on mechanistic interpretability, showing that LLMs are not black boxes and that circuit tracing can reveal multi-step reasoning and human-identifiable concepts.
Belief or Circuitry? Causal Evidence for In-Context Graph Learning
This paper investigates whether LLMs learn in-context through latent structure inference or local pattern matching, using mechanistic interpretability methods like PCA and activation patching on a graph random-walk task.
A Definition of Good Explanations and the Challenges Explaining LLM Outputs
This paper proposes a definition of good explanations based on counterfactuals and prior beliefs, and discusses the inherent difficulties in explaining LLM outputs under this definition.