TokenScope: Token-Level Explainability and Interpretability for Code-Oriented Tasks in Large Language Models

# TokenScope: Token-Level Explainability and Interpretability for Code-Oriented Tasks in Large Language Models
Source: [https://arxiv.org/html/2607.01235](https://arxiv.org/html/2607.01235)
\(5 June 2009\)

###### Abstract\.

Understanding how Large Language Models \(LLMs\) make token\-level decisions during code generation remains a major challenge for both researchers and practitioners\. While recent tools provide insights into model internals or generation outcomes, they often lack decoding\-time signals, fine\-grained uncertainty measures, and interactive mechanisms for exploring alternative generation paths\. We present TokenScope, an interactive interpretability and analysis tool for decoder\-based LLMs that exposes token\-level metrics, attention patterns, and structural information during generation\. TokenScope supports interactive token replacement, counterfactual branching, and code\-aware aggregation via abstract syntax trees\. By unifying decoding\-time signals with structural program analysis, TokenScope enables systematic investigation of LLM behaviour during code generation\.

Large Language Models, Code Generation, Interpretability, Explainability, Attention Analysis, Program Analysis

††copyright:acmlicensed††journalyear:2018††doi:XXXXXXX\.XXXXXXX††conference:Companion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering; June 5–9, 2026; Montreal, Canada††isbn:978\-1\-4503\-XXXX\-X/2018/06††ccs:Computing methodologies Natural language generation††ccs:Computing methodologies Information extraction††ccs:Computing methodologies Lexical semantics††ccs:Computing methodologies Causal reasoning and diagnostics††ccs:Software and its engineering## 1\.Introduction

Large Language Models \(LLMs\) based on transformer decoders have become central to modern software engineering workflows, supporting tasks such as code generation, refactoring, test synthesis, and program understanding\(Chen,[2021](https://arxiv.org/html/2607.01235#bib.bib1); Liet al\.,[2022](https://arxiv.org/html/2607.01235#bib.bib2); Roziereet al\.,[2023](https://arxiv.org/html/2607.01235#bib.bib3)\)\. Despite their effectiveness, these models remain opaque: practitioners and researchers are often unable to explain*why*a model produced a particular token,*how*uncertain it was at different points in the generation, or*what alternatives*were plausibly considered but discarded\. This lack of explainability is particularly problematic in code generation, where small token\-level deviations can cascade into syntactic errors, semantic bugs, or incorrect API usage\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/x1.png)Figure 1\.Overview of TokenScope’s modular architecture\. The generation server exposes decoder\-only LLMs, the orchestration server orchestrates generation, metrics, and branching, and the web\-based frontend application provides interactive exploration of decoding\-time signals\.Existing interpretability and explainability tools provide partial insight into LLM behaviour but leave important gaps\. Prior work largely emphasizes dataset\-level analysis, post\-hoc evaluation, internal representation inspection, or static attention visualization, often requiring offline analysis and substantial expertise\(Tenneyet al\.,[2020](https://arxiv.org/html/2607.01235#bib.bib4); Nanda and Bloom,[2022](https://arxiv.org/html/2607.01235#bib.bib5); Cooney,[2023](https://arxiv.org/html/2607.01235#bib.bib6); Alammar,[2021](https://arxiv.org/html/2607.01235#bib.bib7); Clarket al\.,[2019](https://arxiv.org/html/2607.01235#bib.bib8)\)\. These approaches rarely expose decoding\-time uncertainty, alternative token candidates, or decision margins, and they typically treat generation as a single outcome rather than as a sequence of interdependent token\-level decisions\. As a result, researchers lack the infrastructure to study where uncertainty arises during generation, how attention and probability mass shift over time, and how local decisions propagate into structural errors in code\.

To address these challenges, we introduceTokenScope, an interactive explainability and interpretability environment for decoder\-based LLMs, with a focus on code generation\. TokenScope instruments the decoding process itself, capturing fine\-grained token\-level signals, including confidence measures, margin confidence, entropy, token surprisal, alternative candidates, and attention weights, at each generation step\. These signals are streamed to an interactive frontend that enables direct inspection and comparison of token\-level decisions\.

TokenScope supports both researchers and practitioners by enabling interactive exploration of LLM behaviour during code generation\. Users can apply modifications at any point in a generation by replacing a token with an alternative candidate or custom text and continuing generation from the modified prefix, allowing rapid exploration of counterfactual outputs and failure modes\. By aligning decoding\-time uncertainty and attention signals with code structure via Abstract Syntax Tree \(AST\) analysis, TokenScope helps practitioners diagnose where confidence degrades, identify structurally fragile regions of generated code, and trace how local token\-level decisions propagate into higher\-level program behaviour\. Together, these capabilities provide a practical environment for debugging, auditing, and understanding LLM\-generated code beyond post\-hoc inspection\.

This paper makes the following contributions:

- •we introduce TokenScope, an interactive tool that exposes decoding\-time token\-level signals for transformer\-based LLMs, with a focus on code generation;
- •we provide integrated support for token\-level uncertainty metrics, attention saliency, and interactive branching for counterfactual generation;
- •we enable code\-aware analysis by aligning token\-level signals with AST entities at multiple levels of granularity; and
- •we present a unified environment that combines explainability, interpretability, and user\-driven exploration, addressing limitations of existing tools that focus on post\-hoc analysis or isolated internal signals\.

## 2\.Background

A range of tools has been proposed to improve the interpretability and explainability of transformer\-based models\. We briefly review the most relevant categories and highlight how TokenScope differs\.

General\-purpose interpretability platforms such as LIT\(Tenneyet al\.,[2020](https://arxiv.org/html/2607.01235#bib.bib4)\)and Phoenix\(Arize\-ai,[2025](https://arxiv.org/html/2607.01235#bib.bib9)\)provide interfaces for dataset inspection, model comparison, and embedding analysis\. While valuable for evaluation and debugging at the dataset level, these tools do not expose decoding\-time token probabilities, alternative candidates, or uncertainty metrics during generation\.

Mechanistic interpretability frameworks such as TransformerLens\(Nanda and Bloom,[2022](https://arxiv.org/html/2607.01235#bib.bib5)\)and sparse autoencoder approaches\(Cooney,[2023](https://arxiv.org/html/2607.01235#bib.bib6)\)enable detailed inspection of attention heads, neurons, and internal activations\. These approaches offer powerful analytical capabilities but typically require offline analysis and substantial technical expertise, and they are not designed for interactive exploration of individual generations\.

Visualization tools such as Ecco\(Alammar,[2021](https://arxiv.org/html/2607.01235#bib.bib7)\)and Attention Analysis\(Clarket al\.,[2019](https://arxiv.org/html/2607.01235#bib.bib8)\)focus primarily on attention weights\. While useful for understanding information flow, attention alone is an incomplete proxy for model confidence or decision\-making, and these tools do not integrate uncertainty measures or structural program analysis\.

## 3\.System Overview

TokenScope is an interactive explainability and interpretability system for decoder\-only transformer LLMs, designed to expose fine\-grained token\-level signals during text and code generation\. The system captures internal model statistics at decoding time and presents them through an interactive web interface that supports inspection, comparison, and counterfactual exploration of generation trajectories\.

TokenScope follows a modular three\-component architecture, enabling support for arbitrary decoder\-only models via custom generation server implementations, as illustrated in Figure[1](https://arxiv.org/html/2607.01235#S1.F1)\.

- •\(i\) The generation server is responsible for model discovery, loading, and low\-level decoding, exposing Hugging Face–compatible decoder\-only models through a minimal generation API\.
- •\(ii\) The orchestration server coordinates generation requests through the generation server, manages user sessions, branching generation states, metric computation and aggregation, and frontend\-facing API endpoints\.
- •\(iii\) The web\-based frontend, implemented in React, provides interactive visualizations and controls for exploring model behaviour\.

Generation is performed incrementally and streamed to the frontend\. At each decoding step, TokenScope records token probabilities, alternative candidates, attention weights, and derived uncertainty metrics, associates them with the generated token, and makes them available for interactive visualization and analysis \(Figure[2](https://arxiv.org/html/2607.01235#S3.F2)\)\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_generation.png)Figure 2\.Output generated by Qwen2\.5 Coder 1\.5B Base using greedy generation, when given ‘def sort\(l\):’ as input\. Grey represents the prompt, while other token colors represent token\-level confidence scores in an increasing order of Red, Orange, Dark Yellow and Black\.
## 4\.Functionality Overview

TokenScope is designed as an interactive environment that exposes and manipulates the generation process of decoder\-based LLMs\. This section provides an overview of its core functionalities\.

### 4\.1\.Generation Modes and Decoding Control

TokenScope supports multiple generation modes, including greedy decoding, temperature\-controlled sampling, top\-k, and nucleus \(top\-p\) sampling\. This enables controlled experiments on how decoding strategies influence reliability and diversity in code generation\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_metrics_chart.png)Figure 3\.Confidence, Margin Confidence, Entropy, Sequence Perplexity and Surprisal metrics for each token in the output sort function\. X and Y axes represent token and metric values, respectively\.
### 4\.2\.Decoding\-time Metrics

During generation, TokenScope records token\-level statistics such as probability, rank among candidates, token surprisal, entropy, and margin confidence, as well as sequence\-level metrics such as sequence perplexity\. These metrics are displayed as color\-coded tokens \(Figure[2](https://arxiv.org/html/2607.01235#S3.F2)\) and plotted across all tokens \(Figure[3](https://arxiv.org/html/2607.01235#S4.F3)\), and can be aggregated over spans or structural units\. Formal definitions and formulas are provided in Section[5](https://arxiv.org/html/2607.01235#S5)\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_rev_attn.png)Figure 4\.Attention mass of subsequent tokens to token ‘*l*’ \(first token on line seven\) in the output sort function per attention head and the average of all attention heads\.
### 4\.3\.Attention Analysis

TokenScope exposes attention weights between each generated token and the prior context at decoding time\. From these weights, attention saliency measures how strongly subsequent generated tokens attend to a given token\. These signals can be visualized at the token level per each attention head \(Figure[4](https://arxiv.org/html/2607.01235#S4.F4)\) or aggregated over syntax\-aware units to analyze long\-range dependencies in code generation\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_alt.png)

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_branching.png)

Figure 5\.Left: Five top alternative tokens for token ‘*return*’ in the output sort function\. Right: Original output \(left leaf node\) and the alternative branch when ‘*return*’ is replaced with token ‘*print*’\.
### 4\.4\.Interactive Branching and Counterfactual Generation

Users can interactively replace any generated token with an alternative candidate \(Figure[5](https://arxiv.org/html/2607.01235#S4.F5), left\) or custom input and continue generation from the modified prefix\. TokenScope maintains a branching structure that records shared prefixes and divergent continuations, enabling systematic exploration of alternative reasoning paths \(Figure[5](https://arxiv.org/html/2607.01235#S4.F5), right\)\.

### 4\.5\.Code\-Aware Structural Analysis\.

Using Tree\-Sitter parsers, TokenScope maps generated tokens to AST nodes such as expressions, statements, and functions\. Token\-level metrics and attention signals can be aggregated over these structures, allowing analysis of uncertainty and saliency at semantically meaningful levels of abstraction\.

## 5\.Metrics Definitions\.

This section defines the core metrics exposed by TokenScope\. Letpt​\(v\)p\_\{t\}\(v\)denote the probability assigned by the model to tokenvvat generation steptt\.

Token Probability and Rank\.The probability of the generated tokenyty\_\{t\}is given by:

We refer to this probability as the*confidence score*of the token\.

Token rank is defined as the position ofyty\_\{t\}when candidate tokens are sorted in descending order of probability\.

Token Surprisal\.Token surprisal measures the unexpectedness of a generated token:

\(2\)S​u​r​p​r​i​s​a​l​\(yt\)=−l​o​g​pt​\(yt\)Surprisal\(y\_\{t\}\)=\-logp\_\{t\}\(y\_\{t\}\)
Intuitively, this metric measures how unexpected a generated token is given the preceding context\.

Margin Confidence\.Margin confidence captures the difference between the most likely and the second most likely tokens\. This metric reflects local decisiveness rather than absolute correctness\.

Entropy\.Token\-level entropy quantifies uncertainty over the candidate distribution:

\(3\)Ht=−∑vpt​\(v\)​l​o​g​pt​\(v\)H\_\{t\}=\-\\sum\_\{v\}p\_\{t\}\(v\)logp\_\{t\}\(v\)
Sequence Perplexity\.Sequence perplexity over a generated sequencey1:Ty\_\{1:T\}is defined as

\(4\)P​e​r​p​l​e​x​i​t​y=e​x​p​\(1T−1​∑t=2T−l​o​g​pt​\(yT\)\)Perplexity=exp\(\\frac\{1\}\{T\-1\}\\sum\_\{t=2\}^\{T\}\-logp\_\{t\}\(y\_\{T\}\)\)
which quantifies the average unpredictability of the full generated sequence, where lower values indicate the model finds the sequence more probable\.

Attention Mass\.For a generated tokentit\_\{i\}, attention mass is defined as the average amount of attention it receives from all subsequently generated tokenstjt\_\{j\}wherej\>ij\>i\. Formally, for each attention headhh,

\(5\)A​Mh​\(ti\)=1N−1​∑j=i\+1NAh​\(j,i\),AM\_\{h\}\(t\_\{i\}\)=\\frac\{1\}\{N\-1\}\\sum\_\{j=i\+1\}^\{N\}A\_\{h\}\(j,i\),
whereAh​\(j,i\)A\_\{h\}\(j,i\)denotes the attention weight from tokentjt\_\{j\}to tokentit\_\{i\}in headhh\.

Intuitively, attention mass measures how much a token continues to influence later decoding decisions\.

## 6\.Code\-Aware Structural Analysis

In Code Analysis Mode, TokenScope aligns generated tokens with AST entities using Tree\-Sitter parsers\.

### 6\.1\.Token–Entity Alignment and Aggregation

LLM token boundaries do not generally align with AST entities, as tokenization is independent of syntactic parsing\. An AST entity is considered a candidate for an LLM token if the token is fully contained within the entity span, overlaps with the entity start, or overlaps with the entity end\.

When multiple entities are candidates for a single token, TokenScope assigns the entity with the highest predefined priority, reflecting semantic importance \(e\.g\., identifiers over delimiters\)\.

### 6\.2\.Code Analysis Modes

To support analysis at different semantic resolutions, the tool provides five AST entity modes, each defining a distinct aggregation level for token\-level metrics and attention signals\.

Token Moderepresents the finest granularity, where each generated token is analyzed independently\. This mode is useful for inspecting localized uncertainty, abrupt probability drops, or attention shifts at specific lexical positions\.

Expression Modegroups tokens into syntactic expressions such as literals, binary operations, function calls, or indexing expressions\. Aggregating metrics at this level highlights uncertainty and attention patterns within semantic units that often correspond to logical decisions in code\.

Statement Modeaggregates tokens belonging to complete statements, including assignments, control\-flow statements, and return statements\. This mode enables comparison of confidence and attention across executable units that affect program behaviour\.

Line Modeaggregates tokens belonging to a single line of code, as determined by source\-level line boundaries after formatting\. This mode enables analysis of uncertainty and attention at a granularity commonly used by developers when reading and debugging code, and supports identifying lines that disproportionately contribute to low\-confidence generations or error propagation\.

Block Modegroups tokens within structural blocks such as function bodies, loops, or conditional branches\. Analysis at this level supports studying how uncertainty and attention evolve across larger scopes and control\-flow regions\.

Together, these five modes allow TokenScope users to move seamlessly between fine\-grained token inspection and higher\-level structural analysis, supporting both detailed debugging and holistic evaluation of generated code\.

## 7\.Limitations

TokenScope currently targets decoder\-only autoregressive models and requires access to token probabilities and attention weights, which limits its applicability to closed or heavily abstracted LLM APIs\. Its code\-aware analysis relies on Tree\-Sitter parsers and is presently limited to Python; while support for additional languages is feasible, structural analysis depends on parser availability and may fail on incomplete or syntactically invalid generations\. Finally, TokenScope is designed as an evaluation and analysis tool rather than a production inference system, and the overhead of extracting, aggregating, and transferring fine\-grained interpretability signals prevents it from matching the performance of deployment\-oriented inference frameworks\.

## 8\.Conclusion

We presented TokenScope, an interactive tool for observing, analyzing, and manipulating token\-level behaviour in decoder\-based LLMs for code generation\. By exposing decoding\-time uncertainty metrics, attention signals, interactive branching, and code\-aware structural aggregation, TokenScope enables analyses that are difficult to perform with existing tools\. TokenScope can support both practical debugging and empirical research on LLM behaviour in software engineering\. TokenScope’s source code is open in order to facilitate further study in this direction\.

## Data Availability

## References

- J\. Alammar \(2021\)Ecco: an open source library for the explainability of transformer language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations,Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p2.1),[§2](https://arxiv.org/html/2607.01235#S2.p4.1)\.
- Arize\-ai \(2025\)Arize\-ai\.External Links:[Link](https://github.com/Arize-ai/phoenix)Cited by:[§2](https://arxiv.org/html/2607.01235#S2.p2.1)\.
- M\. Chen \(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p1.1)\.
- K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. Manning \(2019\)What does bert look at? an analysis of bert’s attention\.InBlackBoxNLP@ACL,Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p2.1),[§2](https://arxiv.org/html/2607.01235#S2.p4.1)\.
- A\. Cooney \(2023\)Sparse autoencoder library\.Note:[https://github\.com/ai\-safety\-foundation/sparse\_autoencoder](https://github.com/ai-safety-foundation/sparse_autoencoder)Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p2.1),[§2](https://arxiv.org/html/2607.01235#S2.p3.1)\.
- Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago,et al\.\(2022\)Competition\-level code generation with alphacode\.Science378\(6624\),pp\. 1092–1097\.Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p1.1)\.
- N\. Nanda and J\. Bloom \(2022\)TransformerLens\.Note:[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p2.1),[§2](https://arxiv.org/html/2607.01235#S2.p3.1)\.
- B\. Roziere, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez,et al\.\(2023\)Code llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p1.1)\.
- I\. Tenney, J\. Wexler, J\. Bastings, T\. Bolukbasi, A\. Coenen, S\. Gehrmann, E\. Jiang, M\. Pushkarna, C\. Radebaugh, E\. Reif, and A\. Yuan \(2020\)The language interpretability tool: extensible, interactive visualizations and analysis for NLP models\.Association for Computational Linguistics\.External Links:[Link](https://www.aclweb.org/anthology/2020.emnlp-demos.15)Cited by:[§1](https://arxiv.org/html/2607.01235#S1.p2.1),[§2](https://arxiv.org/html/2607.01235#S2.p2.1)\.

## Appendix ATool Availability

TokenScope is fully open\-source\. The source code is publicly available at[https://github\.com/Amirresm/tokenscope](https://github.com/Amirresm/tokenscope)\. To support reproducibility and ease of deployment, the repository includes a*Dockerfile*that enables consistent setup across environments\.

## Appendix BScreencast

## Appendix CTool Walkthrough

This section provides an illustrative walkthrough of TokenScope for a simple code generation scenario\. In this scenario, the base variant of Qwen 3 0\.6B model is used to continue the inputdef sort\(l\):, effectively generating a sort function in a code completion setting\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_home.png)Figure 6\.Users choose the model, prompt and generation settings on the landing page\.### C\.1\.Landing Page

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_llm_modal.png)

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_gen_setting.png)

Figure 7\.Left: The model management modal displays the currently loaded and all available models\. Right: The generation settings drop\-down allows defining the number of alternative tokens, sampling settings and attention configuration\.Figure[6](https://arxiv.org/html/2607.01235#A3.F6)presents the home page\. User starts by loading a model \(left on Figure[7](https://arxiv.org/html/2607.01235#A3.F7)\), entering a prompt, configuring the generation settings \(right on Figure[7](https://arxiv.org/html/2607.01235#A3.F7)\) and selecting thegeneratebutton\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_gen.png)Figure 8\.The main content view displays the generated output and a high\-level summary of the current generation metrics and metadata\.
### C\.2\.Main Content View

When generation starts, the user is redirected to the main content view \(Figure[8](https://arxiv.org/html/2607.01235#A3.F8)\), where the generated output and a high\-level summary of the current generation metrics and metadata are visible\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_trends.png)Figure 9\.Trends chart displays an overview of all metrics for each token across the generation\.Selecting theView Trends Charton the*Statistics*sidebar opens the trends chart \(Figure[9](https://arxiv.org/html/2607.01235#A3.F9)\), which provides an overview of all metrics per token across the generation\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_token_details.png)Figure 10\.The Token Details sidebar presents special token types, token metrics and alternative tokens, and allows for creating new branches in the generation\.Selecting an individual token switches to theToken Detailssidebar \(Figure[10](https://arxiv.org/html/2607.01235#A3.F10)\), presenting the confidence score and the token’s string \(with whitespace characters such as new line visualized\), special token types, token metrics and alternative tokens\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_attn.png)Figure 11\.The Attention Details sidebar displays tokens’ attention\-relevant metrics, attention source tokens and subsequent attending tokens\.![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_attn_bar.png)Figure 12\.When a target attention token is selected, the user can plot the attention amount of any attention source token for each head\.![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_rev_attn.png)Figure 13\.Heatmap plot of subsequent attending tokens per attention head for the target token\.With an individual token selected, the user can switch to theAttention Detailssidebar \(Figure[11](https://arxiv.org/html/2607.01235#A3.F11)\) to examine the token’s attention\-relevant metrics such as*attention mass*, attention source tokens \(Figure[12](https://arxiv.org/html/2607.01235#A3.F12)\) and subsequent attending tokens \(Figure[13](https://arxiv.org/html/2607.01235#A3.F13)\)\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_branch.png)Figure 14\.Graph view provides an overview of all branches of the generation and a summary of each branch\.
### C\.3\.Generation Branching

When an alternative token is selected in the*Token Details*sidebar, a branch in the generation is created, and the generation is continued with the new prefix, which is managed in theGraph View, as illustrated in Figure[14](https://arxiv.org/html/2607.01235#A3.F14)\.

![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_ast.png)Figure 15\.Code Analysis view displays extracted code entities and their average confidence scores\.![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_ast_bar.png)Figure 16\.Plot view of the average confidence scores across the extracted code entities\.![Refer to caption](https://arxiv.org/html/2607.01235v1/fig_wt_ast_attn.png)Figure 17\.Heatmap of attention relationship between code entities for each attention head\.
### C\.4\.Code Analysis

After selecting a region of the generation corresponding to a code snippet, TokenScope analyses, extracts, and assigns code entities to the generated tokens\. These entities are presented in theCode Analysisview, shown in Figure[15](https://arxiv.org/html/2607.01235#A3.F15)\. TheCode Analysis Statisticsidebar presents a summary of extracted entities and their average confidence scores\. This summary is also provided in a bar plot format \(Figure[16](https://arxiv.org/html/2607.01235#A3.F16)\)\.

Additionally, TokenScope extracts attention relationships between code entities and presents them as a heatmap, as shown in Figure[17](https://arxiv.org/html/2607.01235#A3.F17)\.