What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs

# What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs
Source: [https://arxiv.org/html/2605.28823](https://arxiv.org/html/2605.28823)
Mohamed Abdelwahab44footnotemark:4Michelle Yu CollinsSihan ChenYi Cheng Zhao Zafarullah MahmoodJiading ZhuSoliman AliJonathan Rose The Edward S\. Rogers Sr\. Department of Electrical and Computer Engineering University of Toronto

###### Abstract

As the influence of LLMs expands, it is imperative to gain insight into their decisions\. One way to do that is to develop probes that detect the presence or absence of a broad set of concepts within the embeddings computed in an LLM \- which is what we might say a model is “thinking" about\. Such probes should be low\-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation\.

In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful*delineation*of a concept through the creation of a dataset with the concept both present and then absent\. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed\. Finally, we show that such probes can track concepts across larger contexts\. This is done with four separate concepts and three different LLMs\. When this process is scaled to many more concepts, it will create the ability to easily monitor new models\.

What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs

Mohamed Abdelwahab44footnotemark:4Michelle Yu Collins Sihan Chen Yi Cheng ZhaoZafarullah MahmoodJiading ZhuSoliman AliJonathan RoseThe Edward S\. Rogers Sr\. Department of Electrical and Computer EngineeringUniversity of Toronto

$\\mathsection$$\\mathsection$footnotetext:Corresponding author:
[mo\.abdelwahab@mail\.utoronto\.ca](https://arxiv.org/html/2605.28823v1/mailto:[email protected])## 1Introduction

Large Language Models \(LLMs\) appear to function as*concept machines*, in that they infer implied concepts \(at multiple levels of abstraction\) from their input that are then the driving force behind the generated output\. Indeed, it is now well understood that many kinds of concepts can be detected in internal embeddings within LLMs using linear probes\(Conneauet al\.,[2018](https://arxiv.org/html/2605.28823#bib.bib24); Hupkes and Zuidema,[2018](https://arxiv.org/html/2605.28823#bib.bib25)\)\. These concepts include Parts\-of\-Speech tags, verb tense\(Liuet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib33); Ravishankaret al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib34); Arpset al\.,[2022](https://arxiv.org/html/2605.28823#bib.bib35); Hewitt and Manning,[2019](https://arxiv.org/html/2605.28823#bib.bib36); Durraniet al\.,[2020](https://arxiv.org/html/2605.28823#bib.bib30); Tenneyet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib31); Kimet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib32)\)as well as time, physical location\(Gurnee and Tegmark,[2024](https://arxiv.org/html/2605.28823#bib.bib37)\), and truth\(Burnset al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib38)\)\. There are many other important concepts whose presence or absence needs to be identified, and so the goal in this work is to prototype the steps needed to create a broad set of probes that could be used to monitor an LLM\.

The first step is to select a set of concepts that are needed for the downstream task of monitoring\. In these first steps, we select the general goal of human activities, and \(somewhat arbitrarily\) choose the concepts of ambition, investigation, democracy, and envy\. The next step is to create a working definition of the term*concept*itself, based on the prototype theory\(Ren and Wei,[2019](https://arxiv.org/html/2605.28823#bib.bib73)\)of concepts: a concept is an entity characterized by a set of features used to determine membership within it\. Finally, we use the term ‘delineation’ of a concept to mean the method by which we create labeled examples considered to embody the concept and those that do not\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/ambition_intro_example.jpg)Figure 1:Ambitionprobe output for every word in an expanding context in layer 13 ofLlama\-3\-8BTo illustrate the use of these probes for monitoring, we show how they can track the waxing and waning of concepts in an LLM’s embeddings as more words are added to the input context\. For example, Figure[1](https://arxiv.org/html/2605.28823#S1.F1)illustrates this using a probe trained to detectambitionin theLlama\-3\-8Bmodel\. The probe is applied to the final embedding of an expanding input sequence, taken from the13t​h13^\{th\}transformer layer\. The X\-axis gives the input tokens \(as complete words\), and the Y\-axis gives the probe’s sigmoid output computed on the embedding*after*each new word is added to the sequence\. We take a probe output above 0\.5 to indicate the presence of the concept, while an output below 0\.5 indicates its absence\. The colored shading in the figure indicates the label of the*entire*sentence: green for the presence of ambition and red for its absence\. The sentences in the figure are drawn from the middle of a three\-paragraph story, and it is instructive to read them and compare them with the probe’s outputs\. We observe that the probe output rises above 0\.5 when the newly added span of words implies ambition, and falls below 0\.5 once the continuing context no longer does\.

Previous work has explored the detection of concepts in LLMs, including the use of Sparse Autoencoders \(SAEs\)\(Cunninghamet al\.,[2023](https://arxiv.org/html/2605.28823#bib.bib43); Brickenet al\.,[2023](https://arxiv.org/html/2605.28823#bib.bib20); Lieberumet al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib22); Rajamanoharanet al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib44); Templetonet al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib21)\)to detect a large number of concepts in an LLM\. This is done using unsupervised training of an SAE that is tasked with reconstructing an internal embedding from a large \(enforced\) sparse representation\. This approach creates many concept detectors, but does not have anything equivalent to our stage of*delineation*where the specific concept is chosen and all others are excluded\. SAEs do not permit control over the concepts that can be detected, which is very problematic, especially in matters of trust\. In addition, training is very computationally expensive and must be repeated for every new model\. By contrast, our approach builds a delineating dataset only once per concept, enabling inexpensive probe training on*any model*thereafter\.

Another approach to concept detection, proposed in\(Zouet al\.,[2023](https://arxiv.org/html/2605.28823#bib.bib19)\), allows direct specification of a concept by explicitly instructing the LLM to identify a named concept\. However, this makes it unsuitable for continuous monitoring, as it requires a separate LLM invocation for detection\.

In this work, we create probes for four implied concepts –ambition,investigation,democracy, andenvy– by constructing a dataset with validated presence/absence labels for each\. These labels serve as ground truth for detecting the concepts within the LLM embeddings\. This method requires significantly less training than SAEs and allows direct specification of the concepts to study\.

The primary contributions of this paper are: \(1\) We propose a semi\-automatic LLM\-based method for delineating a concept through the creation of a binary dataset of textual examples for a specified concept\. The method seeks to inhibit patterns that may unintentionally give away the labels; \(2\) We illustrate the utility of the delineation method by training linear probes on the binary datasets and show that these achieve good accuracy\. \(3\) We show how the size of the probes can be constrained to fewer than 80 parameters while still achieving good accuracy\. \(4\) We show the probes used to observe concepts waxing and waning in the model embeddings as words are added to the input context, illustrating their use as low\-cost continuous monitors; \(5\) We provide the created datasets for others to use\. Since each dataset can be reused on any LLM to build probes, we believe this motivates a larger effort to select and delineate many such reusable concept datasets to advance research in LLM explainability\.

## 2Background and Related Work

### 2\.1Concepts in LLMs

Several works explore the existence of concepts in LLMs\.Shaniet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib17)\)andLiaoet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib18)\)investigate LLMs’ knowledge of concept hierarchies\.Shaniet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib17)\)use zero\-shot prompting\(Brownet al\.,[2020](https://arxiv.org/html/2605.28823#bib.bib1)\), directly asking the model to generate an answer on whether one concept is within the broader category of another\. In contrast,Liaoet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib18)\)present the model with statements expressing conceptual relationships and probe its embeddings to assess the validity of those relations\.

Zouet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib19)\)explore concepts in LLMs by extracting embeddings after prompting the model to identify a specific concept in an input example\. They apply PCA\(Pearson,[1901](https://arxiv.org/html/2605.28823#bib.bib26); Hotelling,[1933](https://arxiv.org/html/2605.28823#bib.bib27)\)to these embeddings to derive a ‘concept vector\.’ To detect the concept in new inputs, they use the same prompt to extract an embedding and measure its alignment with the concept vector using dot product\. However, this approach is unsuitable for continuous concept monitoring, since detection requires a separate LLM invocation\.

Cunninghamet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib43)\); Brickenet al\.\([2023](https://arxiv.org/html/2605.28823#bib.bib20)\); Lieberumet al\.\([2024](https://arxiv.org/html/2605.28823#bib.bib22)\); Rajamanoharanet al\.\([2024](https://arxiv.org/html/2605.28823#bib.bib44)\); Templetonet al\.\([2024](https://arxiv.org/html/2605.28823#bib.bib21)\)use SAEs to disentangle LLM embeddings into many dimensions corresponding to individual concepts\. SAEs undergo unsupervised training, so concepts are identified post hoc using an automated LLM\-based method\(Billset al\.,[2023](https://arxiv.org/html/2605.28823#bib.bib29)\)\. As such, the extracted concepts cannot be specified in advance, offering no guarantee of cross\-model consistency, and SAEs demand large\-scale training \(\>1B examples per SAE\)\. There is also no guarantee of disentanglement\. As a result, SAEs are not suitable for dedicated investigations into LLMs’ inference of concepts\.

Our work proposes a probing\-based approach to perform concept detection in LLMs using datasets constructed for specific concepts\. This method enables exploration of these concepts across models, requires far less training than SAEs, and minimizes the risk of cueing the model toward the target concept\. It also enables low\-cost monitoring of how concepts wax and wane in a model’s embeddings as the context expands during generation\.

### 2\.2Linear LLM Probes

LLM probes have been used to study the properties an LLM has acquired during training by using a separate model that makes a prediction given an LLM embedding\. The probe is usually a classifier model that is trained to detect a specific property\. If the probe achieves a reasonable accuracy, it suggests that the property is encoded within the LLM\.

Probes were first applied to early transformer models, such as BERT\(Devlinet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib4)\), to explore whether they encoded linguistic properties such as Parts\-of\-Speech tags and main verb tense\(Liuet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib33); Ravishankaret al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib34); Arpset al\.,[2022](https://arxiv.org/html/2605.28823#bib.bib35); Hewitt and Manning,[2019](https://arxiv.org/html/2605.28823#bib.bib36); Durraniet al\.,[2020](https://arxiv.org/html/2605.28823#bib.bib30); Tenneyet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib31); Kimet al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib32)\)\. More recently, probes have been used to explore the encoding of time, space, and truth within LLMs\(Gurnee and Tegmark,[2024](https://arxiv.org/html/2605.28823#bib.bib37); Burnset al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib38)\)\.

## 3Detection and Tracking of Concepts within LLMs

We use linear probe classifiers to detect concepts in LLMs, and to illustrate how these concepts wax and wane in a model’s embeddings as the input context expands\. In this way, they can be used for monitoring the LLMs for “thinking” about a specific concept\.

### 3\.1Inference of Concepts in LLMs

The training of a probe requires a binary dataset of textual examples for the concept\. The concept should be*present*in positive examples and*absent*in negative ones\. These examples are fed into an LLM to extract embeddings, which are then used to train and evaluate a probing classifier\.

#### 3\.1\.1Concept Dataset Creation

Table 1:Example\-pair created forAmbitiondataset using an example templateWhen creating a binary dataset for a concept, the goal is to ensure that the positive and negative examples differ solely in the presence or absence of the concept, without unintentionally including any patterns that could “give away” the label\(Geirhoset al\.,[2019](https://arxiv.org/html/2605.28823#bib.bib47); Xiaoet al\.,[2021](https://arxiv.org/html/2605.28823#bib.bib46); Wang and Wang,[2024](https://arxiv.org/html/2605.28823#bib.bib45)\)\. This is achieved by using textual example templates that were created independently of any particular concept\. We then generate a pair of examples following the linguistic structure of the template, where the concept is present in one example and absent in the other\. Table[1](https://arxiv.org/html/2605.28823#S3.T1)shows an example template and a positive\-negative example pair generated from it; additional examples are provided in Appendix[A](https://arxiv.org/html/2605.28823#A1)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/concept_dataset_creation.png)Figure 2:Creation of a concept dataset while limiting label leakageWe obtain templates from a dataset based on Project Gutenberg\(Faysse,[n\.d\.](https://arxiv.org/html/2605.28823#bib.bib54); Project Gutenberg,[n\.d\.](https://arxiv.org/html/2605.28823#bib.bib51)\), a free eBook library\. Paragraphs from English books are split into sentences, and templates are formed by randomly selecting one to three consecutive sentences\. We remove templates with incomplete sentences, misplaced words or numbers, or those not centered on human subjects, since we focus on human\-related concepts in this work\. A prompted LLM111We usegpt\-4o\-2024\-08\-06\(OpenAI,[2024](https://arxiv.org/html/2605.28823#bib.bib50)\)for all prompted generators and classifiers\.is used for filtering\. The prompt used, as well as all other prompts employed throughout this paper, are provided in Appendix[B](https://arxiv.org/html/2605.28823#A2)\. We created a set of 30,000 templates, which can be reused to create any specific concept dataset\.

To create a dataset for a specific concept, the concept is first defined as having specific features, following the definition of*concept*in Section[1](https://arxiv.org/html/2605.28823#S1)\. This definition is then provided to two instances of the same LLM: one prompted to generate positive \(concept present\) examples, and the other to generate negative \(absent\) examples\. The prompts instruct the model to mimic the structure of the given example templates while altering the semantics to reflect the presence or absence of the concept\. We diversify the generated examples by instructing the LLM to generate the examples within a specific context that is changed every five examples\. The contexts include*workplace*,*academia*,*sports*,*entrepreneurship*,*politics*,*arts*,*music*,*community*,*science*,*technology*, and*social media*\. Details on the prompt refinement are provided in Appendix[B](https://arxiv.org/html/2605.28823#A2)\.

Because the generated examples may not reliably match their intended labels, and manual verification at scale is impractical, a prompted LLM\-based classifier is used to re\-label all examples for greater reliability\. The classifier is instructed to follow the concept definition, and the prompt is iteratively refined until it exceeds 90% accuracy on at least 360 manually labeled examples\. During manual labeling, human annotators also follow the concept definition\. Appendix[B](https://arxiv.org/html/2605.28823#A2)details the labeling process and inter\-rater reliability\. Figure[2](https://arxiv.org/html/2605.28823#S3.F2)illustrates the generation and labeling workflow\.

To ensure a balanced dataset, we filter it to retain only example pairs with opposite classifier\-assigned labels\. This prevents the probe from exploiting structural similarities between examples that share the same label\.

Four datasets were created using this method for the concepts ofambition\(with 11,854 examples\),investigation\(8,296\),democracy\(10,270\), andenvy\(15,350\)\. The specific definitions of these concepts are provided in Appendix[B](https://arxiv.org/html/2605.28823#A2)\. These concepts were selected because they span various concept categories, as discussed in Appendix[C](https://arxiv.org/html/2605.28823#A3)\. A 70/10/20 train/validation/test split is used for probe training and evaluation\.

We recognize that one might argue that an LLM\-prompted classifier, which labels the presence or absence of a concept, could directly be used for concept monitoring\. This is not practical for two reasons: first, it is a computationally expensive monitor that would not be practical to use for many monitors\. Second, the prompted classifier would only be inferring the ‘thoughts’ of the target model from its language, and not the actual concepts the target was traversing, such as when the model is trying to mislead\(Greenblattet al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib74)\)\.

#### 3\.1\.2Probe Training for a Concept

To train a probe for a specific concept, each example in the dataset is input into the LLM\. The embeddings are extracted from every layer of the model to explore where the concept may appear\. To clarify the extraction process of the embeddings, we formalize it as follows: A textual example is tokenized into a sequenceX=\{x1,x2,…,xN\}X=\\\{x\_\{1\},x\_\{2\},\\dots,x\_\{N\}\\\}, wherexix\_\{i\}denotes theii\-th token andNNis the total number of tokens\. Each token is mapped to an embeddingei0∈ℝdm​o​d​e​le^\{0\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{model\}\}via the embedding layer, wheredm​o​d​e​ld\_\{model\}is the LLM’s embedding size\. These embeddings are then processed through the model’s transformer layers, with layerℓ\\ellproducing the setEℓ=\{e1ℓ,…,eNℓ\}E^\{\\ell\}=\\\{e^\{\\ell\}\_\{1\},\\dots,e^\{\\ell\}\_\{N\}\\\}\.

We train the probe using one of two vectors for each layerℓ\\ell: \(1\) theNt​hN^\{th\}embeddingeNℓe^\{\\ell\}\_\{N\}, which encodes information for the entire context up till tokenNN, or \(2\) the average of all embeddings inEℓE^\{\\ell\}\. We refer to either as the*representative embedding*\.

After obtaining a representative embedding for each example in the labeled dataset, it is assigned the label of the corresponding input text\. The training split of this \(embedding, label\) dataset is used to train a linear probe\(Conneauet al\.,[2018](https://arxiv.org/html/2605.28823#bib.bib24); Hupkes and Zuidema,[2018](https://arxiv.org/html/2605.28823#bib.bib25)\)–a binary linear classifier–to predict the presence or absence of the concept\. The probe, trained via gradient descent, is evaluated on the test set\. It containsdm​o​d​e​ld\_\{model\}parameters\.

A common criticism of probing is that the probe may learn the target feature*itself*rather than detect its existence in the model’s embeddings\(Hewitt and Liang,[2019](https://arxiv.org/html/2605.28823#bib.bib40); Kunz and Kuhlmann,[2020](https://arxiv.org/html/2605.28823#bib.bib41); Zhu and Rudzicz,[2020](https://arxiv.org/html/2605.28823#bib.bib42)\)\. To test for this, the literature suggests control tasks such as training probes on randomized data\. Here, we randomize the training and validation sets–either the embeddings222We randomize embeddings by randomizing the input tokens to the LLM\.or the labels–and then evaluate the probe on the original test set\. A sharp accuracy drop with randomized embeddings indicates that the probe depends on information encoded in the embeddings\. Likewise, a drop with randomized labels suggests that the probe’s performance relies on a meaningful mapping between embeddings and labels, rather than a superficial one\.

Another control task involves reducing the probe’s learning capacity by reducing its number of parameters\. To do so, we apply PCA to reduce the dimensionality of the embeddings\(from 1 up todm​o​d​e​ld\_\{model\}\)333Using principal components derived from the embeddings of the example templates\., thereby reducing the parameters of the linear probe, which are equal to the embedding size\. The probe’s accuracy is evaluated at each dimensionality level; if it remains largely unchanged despite substantial compression, this indicates that the embeddings encode the feature, not the probe\.

### 3\.2Waxing and Waning of Concepts

In Section[4\.1\.2](https://arxiv.org/html/2605.28823#S4.SS1.SSS2)below, we confirm that LLMs can infer concepts\. We were also interested to see if the same trained probes could be used to examine whether and how a concept waxes and wanes in the model embeddings as the input context grows\. This analysis requires longer\-text datasets to which the probes are applied\. These will be referred to as the*Story Datasets*\.

#### 3\.2\.1Story Dataset Creation

For each concept, a dataset of three\-paragraph stories is constructed\. Each paragraph contains 10 sentences, and each pair of paragraphs is connected by a transition sentence, totaling 32 sentences per story\. The studied concept is present only in the transition sentences and is absent elsewhere\. The stories are generated using a prompted LLM\.

The generator prompt defines the target concept and instructs the LLM to generate a story in which the concept is initially absent\. It then instructs the model to insert transition sentences between the paragraphs, each having the concept present\. The prompt also emphasizes maintaining semantic coherence throughout\. Appendix[D](https://arxiv.org/html/2605.28823#A4)provides further details on the generation process\.

The prompted classifier, described in Section[3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1), is used to label each sentence in the story individually, retaining only stories where the concept appears only in the transitions\. A total of 50 stories were created for each target concept\.

#### 3\.2\.2Probing for Waxing and Waning

Each story is input to the LLM and, for each word, we obtain a representative embedding which encodes the input context*up to and including*that word\. Let the story consist of words \(including punctuation\)W=\{w1,…,wS\}W=\\\{w\_\{1\},\\dots,w\_\{S\}\\\}, tokenized into*subword*tokensX=\{x1,…,xS′\}X=\\\{x\_\{1\},\\dots,x\_\{S^\{\\prime\}\}\\\}, whereS′≥SS^\{\\prime\}\\geq S\. At each layerℓ\\ell, the LLM produces embeddingsEℓ=\{e1ℓ,…,eS′ℓ\}E^\{\\ell\}=\\\{e^\{\\ell\}\_\{1\},\\dots,e^\{\\ell\}\_\{S^\{\\prime\}\}\\\}\. The*representative embedding*for wordwiw\_\{i\}will either be the embedding of its final subword token,ei′ℓe^\{\\ell\}\_\{i^\{\\prime\}\}, or the cumulative mean of all embeddings up toei′ℓe^\{\\ell\}\_\{i^\{\\prime\}\}\.

The representative embeddings in each layer are classified by the corresponding probe for the target concept\. This yields a probe output vectorP=\{p1,…,pS\}P=\\\{p\_\{1\},\\dots,p\_\{S\}\\\}, where eachpi∈\[0,1\]p\_\{i\}\\in\[0,1\]indicates the concept’s presence \(above 0\.5\) or absence \(below 0\.5\) in the embedding that encodes the context up to and including wordwiw\_\{i\}\. We can observe these word\-level outputs and compare them to the sentence\-level labels of the story, as shown in Figure[1](https://arxiv.org/html/2605.28823#S1.F1)\. This allows us to assess how the LLM’s encoding of the concept evolves as the story unfolds, and whether the probe output aligns with the sentence labels\.

## 4Experiments and Results

### 4\.1Inference of Implied Concepts

#### 4\.1\.1Experimental Setup

To explore concept inference in LLMs, we tested seven open\-source models from three model families:Llama\-3\-8B\(Grattafioriet al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib23)\),Gemma\-2\(2B,9B\)\(Riviereet al\.,[2024](https://arxiv.org/html/2605.28823#bib.bib7)\), andQwen2\.5\(0\.5B,1\.5B,3B,7B\)\(Yanget al\.,[2025](https://arxiv.org/html/2605.28823#bib.bib62)\)\. All models were accessed through the Hugging Face Transformers library\(Wolfet al\.,[2020](https://arxiv.org/html/2605.28823#bib.bib39)\)\. The details for these models are given in Appendix[E](https://arxiv.org/html/2605.28823#A5)\.

The concepts under test areambition,investigation,democracy, andenvy, which are bolded throughout to avoid ambiguity\. Datasets were created for each, as described in Section[3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1)\.

These datasets were used to produce representative embeddings444Using a single Nvidia A100 GPU with 40GB VRAM\(described in Section[3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2)\) which were then used to train 5 probes \(with different seeds\) per layer, averaging their results\. For the representative embedding, we experimented with both theNt​hN^\{th\}embedding and the mean of all embeddings on a given layer\. Appendix[F](https://arxiv.org/html/2605.28823#A6)describes the probes’ hyperparameter settings\.

We primarily present results for the concept ofambition, with similar results for the other concepts provided in Appendix[G](https://arxiv.org/html/2605.28823#A7)\. Since all LLMs show similar patterns, we focus onLlama\-3\-8B, reporting results for other models in Appendix[G](https://arxiv.org/html/2605.28823#A7), unless otherwise noted\.Llama\-3\-8Bhas adm​o​d​e​ld\_\{model\}size of 4,096 and 32 transformer layers\.

#### 4\.1\.2Results and Discussion

Figure[3](https://arxiv.org/html/2605.28823#S4.F3)shows a plot of the probe accuracy for each concept under test across all layers ofLlama\-3\-8B\. Here, the probes were trained and tested on theNt​hN^\{th\}embedding on each layer\. The Y\-axis is broken at the bottom to include all data points while maintaining the visibility of trends\. All the results are well above 50% \(except for the embedding layer, which is close to 50%\) providing evidence that the LLM infers these concepts\. We had expected that probing the embedding layer \(layer 0\) would be unsuccessful, as the probe would only see the uncontextualized embedding of the one final token, which is clearly not sufficient to encode a concept\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/meta-llama/Meta-Llama-3-8B/last_embed_vs_concepts.jpg)Figure 3:Concept probe accuracy across layers inLlama\-3\-8Bfor all 4 concepts![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/meta-llama/Meta-Llama-3-8B/last_vs_avg_embed.jpg)Figure 4:Ambitionprobe accuracy forLlama\-3\-8Busing average andNt​hN^\{th\}embeddings vs\. layerFigure[4](https://arxiv.org/html/2605.28823#S4.F4)plots the probe accuracy versus layer number forambitioninLlama\-3\-8B, comparing the two types of representative embeddings described in Section[3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2): the full\-layer average embedding and theNt​hN^\{th\}embedding\. Interestingly, the average embedding outperforms theNt​hN^\{th\}embedding across most layers, with the performance gap narrowing in the deeper layers\.

It is interesting to observe that the average embedding from the embedding layer \(layer 0\) achieves 87% accuracy\. These embeddings are produced during the language model’s training but are typically regarded as*uncontextualized*, in contrast to the*contextualized*outputs of transformer layers\. It seems that concepts can be detected in LLMs with simple contextualization—averaging—much like in a bag\-of\-words model\. While this works for small contexts of a few sentences, we show in Section[4\.2\.2](https://arxiv.org/html/2605.28823#S4.SS2.SSS2)that it does not work for longer contexts, as intuition suggests, and so the model computation is necessary to extract the concept\.

To test whether the probe itself learns the concept, we apply the control tasks described in Section[3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2)\. We first explore probe size reduction in Figure[5](https://arxiv.org/html/2605.28823#S4.F5)which plotsambitionprobe accuracy versus the number of probe parameters\. The probe is applied to theNt​hN^\{th\}embedding in three layers ofLlama\-3\-8B\. Even with only 40 probe parameters, the accuracy is at least 75%, well above random guessing, and exhibits diminishing returns toward 100 parameters, well below the maximum of 4,096 forLlama\-3\-8B\. Similar trends hold for other models of different sizes, summarized in Table[2](https://arxiv.org/html/2605.28823#S4.T2)\. Accuracy drops by roughly 15% with 20 parameters and 10% with 40, while performance gains diminish beyond 80 parameters\.

In the second control task, we randomized either the embeddings or the labels and evaluated the resulting probe accuracy\. In both cases, accuracy dropped to around 50% for all layers in all LLMs \(Appendix[G](https://arxiv.org/html/2605.28823#A7)\)\. This suggests that the probe’s performance depends both on the information encoded in the embeddings and on having a meaningful, rather than superficial, mapping between embeddings and labels\. We believe that this demonstrates that the probes are not learning the concepts themselves\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/meta-llama/Meta-Llama-3-8B/LF_sweep_last.jpg)Figure 5:Ambitionprobe accuracy forLlama\-3\-8Bas a function of probe size- ∙\\bulletAll results are in percentage \(%\)\.
- ∙\\bullet“max” denotes 4,096 forLlama\-3\-8B, 2,304 forGemma\-2\-2B, and 896 forQwen2\.5\-0\.5B\.
- ∙\\bulletstandard deviation for each result≤\\leq1%\.

Table 2:Ambitionprobe accuracy across model families, sizes, layers, and probe sizes

### 4\.2Tracking Concepts across LLM Context

Our secondary goal is to investigate how an encoded concept evolves as the input context expands\. To do so, we use the same probes555We use a single trained probe per model layer per concept, rather than the five probes used in Section[4\.1](https://arxiv.org/html/2605.28823#S4.SS1)\.that were trained for the concepts studied in Section[4\.1](https://arxiv.org/html/2605.28823#S4.SS1)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/meta-llama/Meta-Llama-3-8B/average/prominence_aggregate.jpg)Figure 6:Ambitionprobe outputs versus word index, averaged across 50 stories inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/prominence_amb_inv.jpg)Figure 7:Ambition&Investigationprobe outputs across words using final sub\-word token embeddings inLlama\-3\-8B#### 4\.2\.1Experimental Setup

A set of 50 stories, each consisting of 32 sentences, was created for each concept, as described in Section[3\.2\.1](https://arxiv.org/html/2605.28823#S3.SS2.SSS1)\. Recall that the target concept appears only in the two transition sentences that join three paragraphs\. Our goal is to determine whether the probe can detect the waxing and waning of the concept throughout these stories\.

We evaluated the seven LLMs from Section[4\.1\.1](https://arxiv.org/html/2605.28823#S4.SS1.SSS1)using two representative embeddings for each word: the final sub\-word token embedding, and the cumulative mean embedding, as detailed in Section[3\.2\.2](https://arxiv.org/html/2605.28823#S3.SS2.SSS2)\. The former was paired with the concept\-specific probes trained on theNthN^\{\\text\{th\}\}embedding, while the latter was used with probes trained on the mean embedding, both discussed in Section[3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2)\.

To obtain an aggregate view of the probe’s word\-level behavior \(and, by extension, the model embeddings’ behavior\) across all 50 stories, we average its sigmoid outputs for each word index across stories\. Before averaging, sentences are aligned by position and padded666Padded positions were excluded from the computation\.to equal length to ensure that word positions align across stories\. This process is detailed in Appendix[H](https://arxiv.org/html/2605.28823#A8)\.

As in Section[4\.1](https://arxiv.org/html/2605.28823#S4.SS1), all LLMs showed similar trends across the investigated concepts\. Thus, we present results forLlama\-3\-8bonambition, deferring the rest to Appendix[H](https://arxiv.org/html/2605.28823#A8)\.

#### 4\.2\.2Results and Discussion

Figure[6](https://arxiv.org/html/2605.28823#S4.F6)shows the aggregateAmbitionprobe output across word indices, using both final sub\-word token embeddings and cumulative mean embeddings from layer 13 ofLlama\-3\-8B\. Vertical dotted lines mark sentence boundaries, with the uneven spacing between markers reflecting different sentence lengths\. Green and red backgrounds denote concept presence and absence, respectively\. A 10\-word moving average is applied for smoothing\.

For the final sub\-word token embeddings, the aggregate outputs across words in each sentence show a clear trend: they rise and surpass the 0\.5 classification threshold when the concept is present, and fall below 0\.5 when it is absent\. The higher variation at the start of sentences is expected, as the full semantic meaning is less clear earlier on,*and*due to the padding needed to align these results\. This pattern suggests that the LLM embeddings capture changes in concept presence, which align with those in the input context, even across long contexts\. We illustrate how this behavior changes across LLM layers in Appendix[H](https://arxiv.org/html/2605.28823#A8)\.

In contrast, cumulative mean embeddings lose the waxing and waning pattern\. Averaging over many tokens, most unrelated to the concept, dilutes its presence in sentences where it does exist, obscuring changes in concept presence\. This supports the view that concept inference in LLMs goes beyond a simple bag\-of\-words representation, contrary to what one might conclude from Section[4\.1\.2](https://arxiv.org/html/2605.28823#S4.SS1.SSS2)\.

This method of tracking concept dynamics can be used to simultaneously monitor multiple specified concepts at the same time, as LLM monitoring would require\. Figure[7](https://arxiv.org/html/2605.28823#S4.F7)illustrates this with probes forAmbitionandInvestigation, each tracking its respective concept across the same input context shown on the X\-axis\. The second sentence implies both concepts in different segments, which are detected by their respective probes\. Because training and using probes incur low computational cost, this approach can be scaled to many concepts, potentially revealing consistent associations between the model’s inferred concepts and*output concepts*expressed in its generated text\. These associations could provide a basis for exploring whether inferred concepts causally influence the model’s generation of output concepts, thus advancing LLM explainability and enhancing safety through pre\-emptive control of its outputs\.

## 5Conclusion

In this paper, we demonstrate the ability to monitor the internal representations of an LLM for specific concepts\. We do so by using linear probes to detect four such concepts, and also show how the concepts wax and wane in a model’s embeddings as the context expands\. We present a methodology for the careful delineation of a concept, creating datasets that are designed to inhibit accidental leakage of concept labels\. This enables the creation of high\-quality concept probes\. Appendix[I](https://arxiv.org/html/2605.28823#A9)describes the release of the datasets used in this paper\. While the current approach involves manual annotation for each concept, future work will focus on fully automating dataset creation so that many low\-cost probes can be used in the LLM field\.

In future work, we will use a larger set of concept datasets to explore the use of concept monitoring in safety applications\.

## 6Limitations

For concept dataset creation, we use the same LLM with the same prompt to generate all examples\. While we use a new example template to generate each example pair and introduce diverse contexts in the prompts to encourage variation, the generated examples may still follow the same distribution\. Since the train, validation, and test sets \(for probe training and evaluation\) are drawn from this generated data, test accuracy may be slightly misleading, as it may not fully reflect the probe’s ability to generalize to truly unseen data\.

In the dataset creation method used, a key goal was to inhibit the existence of unintended patterns that “give away” the label\. To achieve this, we instructed the LLM\-based generator to create example pairs matching the sentence structure of their corresponding templates\. While we qualitatively verified this behavior by reviewing several samples, we did not implement a systematic method to ensure adherence across all generated example pairs\. Furthermore, there may still be subtle patterns, beyond sentence structure, that inadvertently leak the label during probe training\.

LLM\-prompted classifiers were used to re\-label the generated examples in our datasets\. Each classifier was validated using between 360 and 600 manually labeled examples for each of the investigated concepts\. The labels were assigned by students, not expert linguists, and some of the examples were quite subjective with respect to the label\. We do not know how much error was introduced into the labeling process as a result, which reduces the reliability of the classifiers\.

Our experiments show that limiting the probe size does not significantly impact performance, and that randomizing the training data leads to a sharp drop in performance, both of which suggest that the probe does not independently learn the task\. However, the distinction between the probe learning on its own and the embeddings encoding the feature/concept is not binary but rather a continuumKunz and Kuhlmann \([2020](https://arxiv.org/html/2605.28823#bib.bib41)\)\. This suggests that some portion of the accuracy may still be attributed to the probe itself\.

Although our results indicate that several LLMs can infer the four studied concepts with nearly similar trends, this behavior may not generalize to all concepts\.

## 7Ethics Statement

This study aims to enhance our understanding of LLMs, enabling better control of their behavior\. Since LLMs can be used for bad ends, that understanding will also aid those who seek to use them in that way\.

All experiments were conducted on a single Nvidia A100 GPU \(40GB VRAM\), totaling approximately 1080 GPU hours\. While uncovering the inner workings of LLMs can lead to more efficient models, the computational resources required for such research also carry an environmental cost\.

Manual labeling of subsets to validate the LLM\-based concept classifiers was performed by graduate students and compensated undergraduate engineering summer interns\. The interns were fairly paid for their work, and those who contributed more broadly to this study are acknowledged as co\-authors\. The graduate students, supported by research funding and part of the group writing this paper, are also listed as co\-authors\.

The example templates are sourced from Project GutenbergProject Gutenberg \([n\.d\.](https://arxiv.org/html/2605.28823#bib.bib51)\), a free eBook library available under a permissive license777https://www\.gutenberg\.org/policy/license\.html\. Although Project Gutenberg was not originally intended for NLP research, its licensing terms permit its use for research purposes\.

Although highly offensive language is unlikely in these texts, we used the better\-profanity Python package\(Thanh,[2020](https://arxiv.org/html/2605.28823#bib.bib55)\)\(version 0\.7\.0\), licensed under MIT, to identify templates containing words flagged as abusive by its developers\. This process marked approximately 1,250 out of 30,000 templates\. A manual review of some flagged examples indicated that most contained only mildly offensive language\.

We use the example templates to generate synthetic datasets with an OpenAI model, following their terms888https://openai\.com/policies/row\-terms\-of\-use/\. These datasets were used to probe the open\-source models:Llama\-3\-8B,Gemma\-2family of models \(2B,9B\), andQwen2\.5family of models \(0\.5B,1\.5B,3B,7B\), adhering to the license policies set by their developers999https://www\.llama\.com/llama3/license/101010https://ai\.google\.dev/gemma/terms111111https://huggingface\.co/Qwen/Qwen2\.5\-7B/blob/main/LICENSE\.

We utilized AI assistants in this work for various tasks\. OpenAI ChatGPTOpenAI \([2022](https://arxiv.org/html/2605.28823#bib.bib52)\)was used for polishing the paper’s text, while both ChatGPT and GitHub CopilotGitHub \([2022](https://arxiv.org/html/2605.28823#bib.bib53)\)were used for code completion and suggestions\.

## References

- APA dictionary of psychology\.Note:Accessed: 2025\-05\-18External Links:[Link](https://www.apa.org/topics/glossary)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.5.4.2.1.1)\.
- D\. Arps, Y\. Samih, L\. Kallmeyer, and H\. Sajjad \(2022\)Probing for constituency structure in neural language models\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 6738–6757\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.502/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.502)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- S\. Bills, N\. Cammarata, D\. Mossing, H\. Tillman, L\. Gao, G\. Goh, I\. Sutskever, J\. Leike, J\. Wu, and W\. Saunders \(2023\)Language models can explain neurons in language models\.Note:[https://openaipublic\.blob\.core\.windows\.net/neuron\-explainer/paper/index\.html](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html)Cited by:[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p3.1)\.
- A\.M\. Borghi, L\. Barca, F\. Binkofski, and L\. Tummolini \(2018\)Varieties of abstract concepts: development, use and representation in the brain\.Philosophical Transactions of the Royal Society B: Biological Sciences373\(1752\),pp\. 20170121\.External Links:[Document](https://dx.doi.org/10.1098/rstb.2017.0121)Cited by:[Appendix C](https://arxiv.org/html/2605.28823#A3.p1.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features/index.html)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p3.1)\.
- T\. Brown, B\. Mann, N\. Ryder, and et al\. \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p1.1)\.
- C\. Burns, H\. Ye, D\. Klein, and J\. Steinhardt \(2024\)Discovering latent knowledge in language models without supervision\.External Links:2212\.03827,[Link](https://arxiv.org/abs/2212.03827)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- N\. Caramelli and A\. Setti \(2005\)Different domains in abstract concepts\.InProceedings of the Annual Meeting of the Cognitive Science Society,Vol\.27\.External Links:[Link](https://escholarship.org/uc/item/57x1c6mw)Cited by:[Appendix C](https://arxiv.org/html/2605.28823#A3.p1.1)\.
- F\. Conca, E\. Catricalà, M\. Canini, A\. Petrini, G\. Vigliocco, S\.F\. Cappa, and P\.A\. Della Rosa \(2021\)In search of different categories of abstract concepts: a fmri adaptation study\.Scientific Reports11\(1\),pp\. 22587\.External Links:[Document](https://dx.doi.org/10.1038/s41598-021-02013-8)Cited by:[Appendix C](https://arxiv.org/html/2605.28823#A3.p1.1)\.
- A\. Conneau, G\. Kruszewski, G\. Lample, L\. Barrault, and M\. Baroni \(2018\)What you can cram into a single $&\!\#\* vector: probing sentence embeddings for linguistic properties\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 2126–2136\.External Links:[Link](https://aclanthology.org/P18-1198/),[Document](https://dx.doi.org/10.18653/v1/P18-1198)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2.p3.1)\.
- H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey \(2023\)Sparse autoencoders find highly interpretable features in language models\.ArXivabs/2309\.08600\.External Links:[Link](https://api.semanticscholar.org/CorpusID:261934663)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p3.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- N\. Durrani, H\. Sajjad, F\. Dalvi, and Y\. Belinkov \(2020\)Analyzing individual neurons in pre\-trained language models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4865–4880\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.395/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.395)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- M\. Faysse \(n\.d\.\)Project gutenberg dataset\.Note:Accessed: 2025\-02\-14External Links:[Link](https://huggingface.co/datasets/manu/project_gutenberg)Cited by:[§3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1.p2.1)\.
- R\. Geirhos, P\. Rubisch, C\. Michaelis, M\. Bethge, F\. A\. Wichmann, and W\. Brendel \(2019\)ImageNet\-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bygh9j09KX)Cited by:[§3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1.p1.1)\.
- GitHub \(2022\)GitHub copilot\.Note:Accessed: 2025\-02\-14External Links:[Link](https://github.com/features/copilot)Cited by:[§7](https://arxiv.org/html/2605.28823#S7.p7.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, and et al\. \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1\.1](https://arxiv.org/html/2605.28823#S4.SS1.SSS1.p1.1)\.
- R\. Greenblatt, C\. Denison, B\. Wright, F\. Roger, M\. MacDiarmid, S\. Marks, J\. Treutlein, T\. Lydecke, J\. Hernandez\-Orallo, E\. Hubinger,et al\.\(2024\)Alignment faking in large language models\.arXiv preprint arXiv:2412\.14093\.External Links:[Link](https://arxiv.org/abs/2412.14093)Cited by:[§3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1.p7.1)\.
- W\. Gurnee and M\. Tegmark \(2024\)Language models represent space and time\.External Links:2310\.02207,[Link](https://arxiv.org/abs/2310.02207)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- J\. Hewitt and P\. Liang \(2019\)Designing and interpreting probes with control tasks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2733–2743\.Cited by:[§3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2.p4.1)\.
- J\. Hewitt and C\. D\. Manning \(2019\)A structural probe for finding syntax in word representations\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4129–4138\.External Links:[Link](https://aclanthology.org/N19-1419/),[Document](https://dx.doi.org/10.18653/v1/N19-1419)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- H\. Hotelling \(1933\)Analysis of a complex of statistical variables into principal components\.\.Journal of Educational Psychology24,pp\. 498–520\.External Links:[Link](https://api.semanticscholar.org/CorpusID:144828484)Cited by:[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p2.1)\.
- D\. Hupkes and W\. Zuidema \(2018\)Visualisation and ’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure \(extended abstract\)\.InProceedings of the Twenty\-Seventh International Joint Conference on Artificial Intelligence, IJCAI\-18,pp\. 5617–5621\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2018/796),[Link](https://doi.org/10.24963/ijcai.2018/796)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2.p3.1)\.
- N\. Kim, R\. Patel, A\. Poliak, P\. Xia, A\. Wang, T\. McCoy, I\. Tenney, A\. Ross, T\. Linzen, B\. Van Durme, S\. R\. Bowman, and E\. Pavlick \(2019\)Probing what different NLP tasks teach machines about function word comprehension\.InProceedings of the Eighth Joint Conference on Lexical and Computational Semantics \(\*SEM 2019\),R\. Mihalcea, E\. Shutova, L\. Ku, K\. Evang, and S\. Poria \(Eds\.\),Minneapolis, Minnesota,pp\. 235–249\.External Links:[Link](https://aclanthology.org/S19-1026/),[Document](https://dx.doi.org/10.18653/v1/S19-1026)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- J\. Kunz and M\. Kuhlmann \(2020\)Classifier probes may just learn from linear context features\.InProceedings of the 28th International Conference on Computational Linguistics,D\. Scott, N\. Bel, and C\. Zong \(Eds\.\),Barcelona, Spain \(Online\),pp\. 5136–5146\.External Links:[Link](https://aclanthology.org/2020.coling-main.450/),[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.450)Cited by:[§3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2.p4.1),[§6](https://arxiv.org/html/2605.28823#S6.p4.1)\.
- J\. R\. Landis and G\. G\. Koch \(1977\)The measurement of observer agreement for categorical data\.\.Biometrics33\(1\),pp\. 159–74\.External Links:[Link](https://api.semanticscholar.org/CorpusID:11077516)Cited by:[§B\.3](https://arxiv.org/html/2605.28823#A2.SS3.p2.1)\.
- J\. Liao, X\. Chen, and L\. Du \(2023\)Concept understanding in large language models: an empirical study\.External Links:[Link](https://openreview.net/forum?id=losgEaOWIL7)Cited by:[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p1.1)\.
- T\. Lieberum, S\. Rajamanoharan, A\. Conmy, L\. Smith, N\. Sonnerat, V\. Varma, J\. Kramar, A\. Dragan, R\. Shah, and N\. Nanda \(2024\)Gemma scope: open sparse autoencoders everywhere all at once on gemma 2\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, N\. Kim, J\. Jumelet, H\. Mohebbi, A\. Mueller, and H\. Chen \(Eds\.\),Miami, Florida, US,pp\. 278–300\.External Links:[Link](https://aclanthology.org/2024.blackboxnlp-1.19/),[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.19)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p3.1)\.
- N\. F\. Liu, M\. Gardner, Y\. Belinkov, M\. E\. Peters, and N\. A\. Smith \(2019\)Linguistic knowledge and transferability of contextual representations\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 1073–1094\.External Links:[Link](https://aclanthology.org/N19-1112/),[Document](https://dx.doi.org/10.18653/v1/N19-1112)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- Merriam\-Webster \(2025a\)Emotion\.Note:Accessed: 2025\-05\-18External Links:[Link](https://www.merriam-webster.com/dictionary/emotion)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.2.1.2.1.1)\.
- Merriam\-Webster \(2025b\)Society\.Note:Accessed: 2025\-05\-18External Links:[Link](https://www.merriam-webster.com/dictionary/society)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.6.5.2.1.1)\.
- OpenAI \(2022\)Introducing chatgpt\.Note:Accessed: 2025\-02\-14External Links:[Link](https://openai.com/index/chatgpt/)Cited by:[§7](https://arxiv.org/html/2605.28823#S7.p7.1)\.
- OpenAI \(2024\)Hello gpt\-4o\.Note:Accessed: 2025\-02\-11External Links:[Link](https://openai.com/index/hello-gpt-4o/)Cited by:[footnote 1](https://arxiv.org/html/2605.28823#footnote1)\.
- Oxford\-Languages \(2025a\)Action\.Note:Accessed: 2025\-05\-18External Links:[Link](https://languages.oup.com/google-dictionary-en/)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.3.2.2.1.1)\.
- Oxford\-Languages \(2025b\)Aesthetic\.Note:Accessed: 2025\-05\-18External Links:[Link](https://languages.oup.com/google-dictionary-en/)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.8.7.2.1.1)\.
- Oxford\-Languages \(2025c\)Attitude\.Note:Accessed: 2025\-05\-18External Links:[Link](https://languages.oup.com/google-dictionary-en/)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.4.3.2.1.1)\.
- Oxford\-Languages \(2025d\)Morality\.Note:Accessed: 2025\-05\-18External Links:[Link](https://languages.oup.com/google-dictionary-en/)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.7.6.2.1.1)\.
- Oxford\-Languages \(2025e\)Number\.Note:Accessed: 2025\-05\-18External Links:[Link](https://languages.oup.com/google-dictionary-en/)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.9.8.2.1.1)\.
- Oxford\-Languages \(2025f\)Self\-concept\.Note:Accessed: 2025\-05\-18External Links:[Link](https://languages.oup.com/google-dictionary-en/)Cited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.10.9.2.1.1)\.
- K\. Pearson \(1901\)LIII\. on lines and planes of closest fit to systems of points in space\.The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science2\(11\),pp\. 559–572\.External Links:[Document](https://dx.doi.org/10.1080/14786440109462720),[Link](https://doi.org/10.1080/14786440109462720),https://doi\.org/10\.1080/14786440109462720Cited by:[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p2.1)\.
- Project Gutenberg \(n\.d\.\)Project Gutenberg\.Note:Accessed: 2025\-02\-11External Links:[Link](https://www.gutenberg.org/)Cited by:[§3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1.p2.1),[§7](https://arxiv.org/html/2605.28823#S7.p4.1)\.
- S\. Rajamanoharan, T\. Lieberum, N\. Sonnerat, A\. Conmy, V\. Varma, J\. Kramár, and N\. Nanda \(2024\)Jumping ahead: improving reconstruction fidelity with jumprelu sparse autoencoders\.arXiv preprint arXiv:2407\.14435\.Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p3.1)\.
- V\. Ravishankar, M\. Gökırmak, L\. Øvrelid, and E\. Velldal \(2019\)Multilingual probing of deep pre\-trained contextual encoders\.InProceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing,J\. Nivre, L\. Derczynski, F\. Ginter, B\. Lindi, S\. Oepen, A\. Søgaard, and J\. Tidemann \(Eds\.\),Turku, Finland,pp\. 37–47\.External Links:[Link](https://aclanthology.org/W19-6205/)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- R\. Ren and L\. Wei \(2019\)The prototype view of concepts\.InRough Sets,T\. Mihálydeák, F\. Min, G\. Wang, M\. Banerjee, I\. Düntsch, Z\. Suraj, and D\. Ciucci \(Eds\.\),Cham,pp\. 166–178\.External Links:ISBN 978\-3\-030\-22815\-6Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p2.1)\.
- M\. Riviere, S\. Pathak, P\. G\. Sessa, and et al\. \(2024\)Gemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[§4\.1\.1](https://arxiv.org/html/2605.28823#S4.SS1.SSS1.p1.1)\.
- S\. P\. SCHWARTZ \(1980\)Natural kinds and nominal kinds\*\.MindLXXXIX\(354\),pp\. 182–195\.External Links:ISSN 0026\-4423,[Document](https://dx.doi.org/10.1093/mind/LXXXIX.354.182),[Link](https://doi.org/10.1093/mind/LXXXIX.354.182),https://academic\.oup\.com/mind/article\-pdf/LXXXIX/354/182/9870690/182\.pdfCited by:[Table C\.1](https://arxiv.org/html/2605.28823#A3.T1.1.11.10.2.1.1)\.
- C\. Shani, J\. Vreeken, and D\. Shahaf \(2023\)Towards concept\-aware large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13158–13170\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.877/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.877)Cited by:[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p1.1)\.
- A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, B\. Chen, A\. Pearce, C\. Citro, E\. Ameisen, A\. Jones, H\. Cunningham, N\. L\. Turner, C\. McDougall, M\. MacDiarmid, C\. D\. Freeman, T\. R\. Sumers, E\. Rees, J\. Batson, A\. Jermyn, S\. Carter, C\. Olah, and T\. Henighan \(2024\)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p3.1)\.
- I\. Tenney, P\. Xia, B\. Chen, A\. Wang, A\. Poliak, R\. T\. McCoy, N\. Kim, B\. V\. Durme, S\. R\. Bowman, D\. Das, and E\. Pavlick \(2019\)What do you learn from context? probing for sentence structure in contextualized word representations\.ArXivabs/1905\.06316\.External Links:[Link](https://api.semanticscholar.org/CorpusID:108300988)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.28823#S2.SS2.p2.1)\.
- S\. N\. Thanh \(2020\)Better profanity: blazingly fast cleaning of swear words \(and their leetspeak\) in strings\.Note:accessed on 2025\-02\-14External Links:[Link](https://pypi.org/project/better-profanity/)Cited by:[§7](https://arxiv.org/html/2605.28823#S7.p5.1)\.
- Y\. Wang and X\. Wang \(2024\)On the effect of key factors in spurious correlation: a theoretical perspective\.InProceedings of The 27th International Conference on Artificial Intelligence and Statistics,S\. Dasgupta, S\. Mandt, and Y\. Li \(Eds\.\),Proceedings of Machine Learning Research, Vol\.238,pp\. 3745–3753\.External Links:[Link](https://proceedings.mlr.press/v238/wang24j.html)Cited by:[§3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1.p1.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. Rush \(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Q\. Liu and D\. Schlangen \(Eds\.\),Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[§4\.1\.1](https://arxiv.org/html/2605.28823#S4.SS1.SSS1.p1.1)\.
- K\. Xiao, L\. Engstrom, A\. Ilyas, and A\. Madry \(2021\)Noise or signal: the role of image backgrounds in object recognition\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gl3D-xY7wLq)Cited by:[§3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1\.1](https://arxiv.org/html/2605.28823#S4.SS1.SSS1.p1.1)\.
- Z\. Zhu and F\. Rudzicz \(2020\)An information theoretic view on selecting linguistic probes\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 9251–9262\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.744/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.744)Cited by:[§3\.1\.2](https://arxiv.org/html/2605.28823#S3.SS1.SSS2.p4.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. Hendrycks \(2023\)Representation engineering: a top\-down approach to ai transparency\.External Links:2310\.01405,[Link](https://arxiv.org/abs/2310.01405)Cited by:[§1](https://arxiv.org/html/2605.28823#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.28823#S2.SS1.p2.1)\.

## Appendix ADataset Examples

The probing approach used in this work requires the creation of a binary dataset for each investigated concept\. To achieve this, we employ example templates to generate example pairs, where one example has the concept present and the other has it absent\. We show samples of generated examples, along with their corresponding templates, for the concepts of*ambition*\(Table[A\.3](https://arxiv.org/html/2605.28823#A1.T3)\),*investigation*\(Table[A\.4](https://arxiv.org/html/2605.28823#A1.T4)\),*democracy*\(Table[A\.5](https://arxiv.org/html/2605.28823#A1.T5)\), and*envy*\(Table[A\.6](https://arxiv.org/html/2605.28823#A1.T6)\)\.

Table A\.3:Example\-pairs created forAmbitiondataset using different example templates\. The concept is present in the positive examples and absent in the negative examples\.Table A\.4:Example\-pairs created forInvestigationdataset using different example templates\. The concept is present in the positive examples and absent in the negative examples\.Table A\.5:Example\-pairs created forDemocracydataset using different example templates\. The concept is present in the positive examples and absent in the negative examples\.Table A\.6:Example\-pairs created forEnvydataset using different example templates\. The concept is present in the positive examples and absent in the negative examples\.
## Appendix BConcept Dataset Creation Details

Generating concept datasets involved three main steps: \(1\) creating a dataset of example templates, \(2\) using these templates to generate positive\-negative example pairs, and \(3\) re\-labeling the generated examples to improve reliability\. We also filtered out examples where the concept was mentioned explicitly rather than implied\.

### B\.1Example Templates Creation

We obtained the example templates from a dataset of eBooks by splitting paragraphs from books to sentences and randomly selecting one to three consecutive sentences to create each template\. We then used the prompt in Box[B\.3](https://arxiv.org/html/2605.28823#A2.SS3)to filter out any templates that had incoherent sentences, misplaced words, or were not focused on human subjects\.

### B\.2Example Pair Generation

To generate example pairs for a concept, we began by drafting a definition that included features we initially believed were essential to the concept\. We then prompted an LLM, without using our draft definition, to generate examples labeled as either having the concept or not\. Each of us then independently re\-labeled the examples, guided by both the draft definition and our intuition\. This process helped us identify implicit features we used in labeling that were missing from the definition, which we then added, while removing irrelevant or unimportant features\. After two to three iterations of this process, we finalized the definition\. We used this definition in both the generation of examples and their labeling\. The definitions developed through this process are listed as follows:

- •Ambition: a character’s desire to achieve a goal, higher status, or result through their efforts, skill, or courage\.
- •Investigation: a systematic process of inquiry or examination conducted to uncover facts, gather information, or solve a problem, typically involving careful observation, analysis, and evaluation of evidence or data to arrive at conclusions or determine the truth about a particular matter\.
- •Democracy: a system of governance in which decision\-making power is vested in the people, either directly or through elected representatives\. It is based on equal rights for everyone, the rule of law \(no one, not even leaders, is above the law\) and the idea that those in power are accountable to the people\.
- •Envy: the feeling of resentment or discontent evoked by another individual’s perceived advantage, which the subject lacks and desires or deems necessary to acquire\.

To generate example pairs, we prompted an LLM\-based generator using the prompt templates in Boxes[B\.3](https://arxiv.org/html/2605.28823#A2.SS3)\(for positive examples\) and[B\.3](https://arxiv.org/html/2605.28823#A2.SS3)\(for negative examples\) to provide instructions and specify the \{context\} in which examples should be generated\. The placeholders \{concept\} and \{concept definition\} are filled with the target concept and its definition, \{num\_examples\} specifies the number of examples to generate, and \{concept\-specific instructions\} includes any additional instructions specific to the investigated concept\. Before running the generator, we appended \{num\_examples\} example templates for the LLM to mimic\. We filled \{concept\-specific instructions\} with additional instructions for each concept, except forinvestigation, as detailed below:

- •Ambition: “The generated examples MUST show this in a positive way \(the example must not convey a lack of ambition\)\.”
- •Democracy: “Try to minimize using keywords, like "vote", "representative", "collective", that make the concept too obvious in the context\.”
- •Envy: “Avoid mentioning words like "envy", "envious", "jealous", or "jealousy" in the examples\.”

We iteratively refined these prompts to ensure that the generated examples mimic the templates\. After each refinement, we generated 50 positive\-negative example pairs, reviewed them, and adjusted the prompt as needed\. This process continued until the examples consistently mirrored the templates\.

### B\.3Dataset Annotation

To strengthen the reliability of the labels for the generated example pairs, we used a concept\-specific LLM\-based classifier to re\-label all generated example pairs, following the prompt template in Box[B\.3](https://arxiv.org/html/2605.28823#A2.SS3)\. We validated the classifier’s performance on a small, manually labeled dataset\. Human labeling was guided by each concept’s definition to determine whether the examples had the concept present\.

Four annotators independently labeled 397 examples forambitionand 500 examples forenvy, indicating whether each concept was present in the text\. Uncertain examples were marked as “borderline\.” We assessed inter\-rater reliability using Cohen’s Kappa for pairwise agreement, shown in Figure[B\.1](https://arxiv.org/html/2605.28823#A2.F1)forambitionand Figure[B\.2](https://arxiv.org/html/2605.28823#A2.F2)forenvy\. We also evaluated the group\-wise agreement using Fleiss’ Kappa, obtaining a value of 0\.75 for both concepts, indicating substantial agreement\(Landis and Koch,[1977](https://arxiv.org/html/2605.28823#bib.bib56)\)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/ambition_kappa.jpg)Figure B\.1:Inter\-rater reliability for examples labeled on the presence/absence ofambition![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/envy_kappa.jpg)Figure B\.2:Inter\-rater reliability for examples labeled on the presence/absence ofenvyWe filtered each dataset to retain only examples with confidently assigned labels, based on one of the following criteria:

- •Perfect agreement on the label \(all annotators assigned the same label\) with less than 2 “borderline” flags
- •Semi\-perfect agreement on the label \(3 versus 1\) with no “borderline” flags
- •Semi\-perfect agreement on the label \(3 versus 1\) with only one “borderline” flag assigned by the annotator who deviated from the majority

Given that we used majority vote to assign the labels to each example, these conditions ensured that even if the annotator marking the example as “borderline” had flipped their label, the overall classification for that example would remain unchanged\.

After filtering, we obtained 360 examples forambitionand 471 forenvy\. Using these datasets, we validated their respective LLM\-based classifiers with the prompt template shown in Box[B\.3](https://arxiv.org/html/2605.28823#A2.SS3)\. The classifier achieved 98% accuracy forambitionand 93% forenvy\.

A single annotator labeled 500 examples forinvestigationand 600 fordemocracy\. Using the same prompt template, the LLM\-based classifiers achieved 92% accuracy for both concepts\.

Box B\.1: Example Templates\-Filter PromptClassify the following example as either "True" or "False" based on the given conditions:\- The text must contain complete and coherent sentences with actionable verbs\.\- The text must be free of out\-of\-place words, numbers \(like chapter titles\), or ISBN numbers\.\- The text should mainly focus on human subjects and their actions/interactions, not on the surrounding environment or non\-human subjects\.classify as "True" only if all of these conditions are met, otherwise, classify as "False"\.Example:Box B\.2: Positive Example Generation PromptGenerate enumerated examples which mimic the provided sentences only in terms of subject\-verb order, but not in semantic meaning\.The semantic meaning should be changed so that the concept of \{concept\} is obvious in the context\.\{concept\} is \{concept definition\}You can add a few more words to the original example length to achieve this, or you can use a slightly fewer number of words\.Do not repeat the ideas in the previously generated examples\.\{Concept\-specific instructions\.\}Do not refer to the characters as ‘‘The \_\_\_’’\.Generate exactly \{num\_examples\} examples based on the given enumerated examples\. Output only the example and its enumeration\. The examples that you generate must be in the context of \{context\}\. Here are the enumerated examples:Box B\.3: Negative Example Generation PromptGenerate enumerated examples which mimic the provided sentences only in terms of subject\-verb order, but not in semantic meaning\.The semantic meaning should be changed so that the context is irrelevant to the concept of \{concept\} whatsoever\.\{concept\} is \{concept definition\}Irrelevance to \{concept\} means not showing these traits in the text, and not even showing the opposite of this\.The context must still be focused on human subjects rather than on the setting or surrounding environment\.You can add a few more words to the original example length to achieve this, or you can use a slightly fewer number of words\.Do not repeat the ideas in the previously generated examples\.Do not refer to the characters as ‘‘The \_\_\_’’\.Generate exactly \{num\_examples\} examples based on the given enumerated examples\. Output only the example and its enumeration\. The examples that you generate must be in the context of \{context\}\. Here are the enumerated examples:Box B\.4: Concept Classification/Re\-labeling PromptClassify the following input as either implying the concept of \{concept\} or not\.\{concept\} is \{concept definition\}If the given input implies \{concept\}, output 1, else output 0\.
### B\.4Filtration of the Created Dataset

To prevent the task from becoming trivial, we filtered each concept\-specific dataset to exclude example pairs where either example contained an explicit mention of the concept or a close synonym\. We removed all forms of such words using their word stems\. The stems used for each concept are:

- •Ambition: “ambit” \(such as*ambition*and*ambitious*\), and “aspir” \(such as*aspire*and*aspiration*\)\.
- •Investigation: “investigat” \(such as*investigate*and*investigation*\), and “examin” \(such as*examine*and*examination*\)\.
- •Democracy: “democra” \(such as*democracy*and*democratic*\)\.
- •Envy: “env” \(such as*envy*and*envious*\), and “jealous” \(such as*jealous*and*jealousy*\)\.

## Appendix CConcept Selection

The concepts selected for this study span multiple categories\. As there is no universally accepted taxonomy, we synthesized one by integrating taxonomies from prior literature\.Concaet al\.\([2021](https://arxiv.org/html/2605.28823#bib.bib57)\)andCaramelli and Setti \([2005](https://arxiv.org/html/2605.28823#bib.bib60)\)both include emotions and cognitive processes in their taxonomy; the former also includes attitudes and human actions, while the latter adds nominal kinds and states of the self\. In contrast,Borghiet al\.\([2018](https://arxiv.org/html/2605.28823#bib.bib58)\)classify concepts into emotions, social, moral, and aesthetic categories, as well as numbers\. We combined these into a unified set of categories, shown in Table[C\.1](https://arxiv.org/html/2605.28823#A3.T1)\. We note that a concept may belong to multiple categories\. The classifications for each concept in our study are as follows:

- •Ambition:*state of self*and*attitudes*
- •Investigation:*cognitive processes*
- •Democracy:*social concepts*
- •Envy:*emotions*

Table C\.1:Categories of Abstract Concepts
## Appendix DDetails on Story Dataset Creation

To investigate whether and how an LLM tracks concept strength, we created story datasets for each concept, with the concept appearing in only two out of 32 sentences\. Each story comprised three 10\-sentence paragraphs where the concept was absent, connected by transition sentences in which the concept was present\.

To create these stories, we used an LLM in a chat setup\. The initial prompt \(prompt template shown in Box[D](https://arxiv.org/html/2605.28823#A4)\) instructed the model to generate three 10\-sentence paragraphs\. The placeholders \{concept\}, \{concept definition\}, and \{context\} were filled with the target concept, its definition, and a specified context, similar to those in Section[3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1)\. After verifying that the story met the required structure, we used a second prompt \(prompt template shown in Box[D](https://arxiv.org/html/2605.28823#A4)\), with the story and original prompt in context, to instruct the LLM to insert single\-sentence transitions between paragraphs\. These transitions were designed to have the concept present while maintaining semantic coherence\. The placeholder \{concept\-related words\} was replaced with a set of terms that would have made the concept explicit, as detailed in the following list:

- •Ambition: “ambition”, “ambitious”, “aspire”, or “aspiration”\.
- •Investigation: “investigation”, “investigate”, “examine”, or “examination”\.
- •Democracy: “democracy”, “democratic”, or “democratize”\.
- •Envy: “envy”, “envious”, “jealous”, or “jealousy”\.

Box D\.1: Story Creation Initial PromptGenerate a 3\-paragraph story, where each paragraph is made up of 10 sentences\.You must abide by that 10\-sentence rule\.The paragraphs must be coherent and logically connected to form a meaningful narrative\.All sentences in those paragraphs must be irrelevant to the concept of \{concept\}\.\{concept\} is \{concept definition\}\.The story must be focused on human subjects, not on the environment or animals\.The story must be in the context of \{context\}\.The story must be written so that later, it can be changed to include the concept of \{concept\}, but the original story you generate must have this concept absent\.Do not number the paragraphs or the sentences within the paragraphs and do not include any special characters to highlight the different paragraphs\.Box D\.2: Story Creation Continuation PromptGiven this story, connect each paragraph to the next one with only one connecting sentence per connection\.Each connecting sentence must be coherent and logically connected to both paragraphs it joins\.The tone of the connecting sentences should match the tone of the story\.The concept of \{concept\} must be obvious in the connecting sentences\.\{concept\} is \{concept definition\}\.The connecting sentences must not include words that make \{concept\} explicit such as \{concept\-related words\}\.You can make very slight modifications to the original story to ensure that the connecting sentences are coherent and logically connected to the story, but the modified sentences must maintain the irrelevance to \{concept\}\.Include the whole story with the connecting sentences in your output, not just the connecting sentences\.Do not include any special characters to highlight the connecting sentences\.After generating each story, we verified that it contained exactly 32 sentences\. We then split the story into individual sentences and re\-labeled them using the LLM\-based classifier from Section[3\.1\.1](https://arxiv.org/html/2605.28823#S3.SS1.SSS1)to ensure that the main paragraphs did not include the concept, while the transition sentences did\. To confirm the concept was not mentioned explicitly, we checked for the presence of its word stems, as detailed in Appendix[B\.4](https://arxiv.org/html/2605.28823#A2.SS4), and excluded any stories where they appeared\.

## Appendix ELLM Details

Table[E\.1](https://arxiv.org/html/2605.28823#A5.T1)provides details for the LLMs evaluated in this work\.

ModelModelNumberEmbedd\-Familysizeof layersing sizeLlama\-38B324,096Gemma\-22B262,3049B423,584Qwen2\.50\.5B248961\.5B281,5363B362,0487B283,584Table E\.1:Details for the studied LLMs
## Appendix FProbe Training Specifications

In our experiments, we used the following settings to train the probes:

- •Optimizer: Adam
- •Learning rate: 0\.005
- •Batch size: 512
- •Number of epochs: 500 \(with early stopping\)
- •Train/validation/test split: 70%/10%/20%

## Appendix GExtended Results for Inference of Concepts

### G\.1Probe Accuracies for all Concepts

Figures[G\.1](https://arxiv.org/html/2605.28823#A7.F1)–[G\.6](https://arxiv.org/html/2605.28823#A7.F6)illustrate the probe accuracies for all concepts across the LLM layers for each model included in this investigation\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/google/gemma-2-2b/last_embed_vs_concepts.jpg)Figure G\.1:Probe accuracies across layers for all concepts inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/google/gemma-2-9b/last_embed_vs_concepts.jpg)Figure G\.2:Probe accuracies across layers for all concepts inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/Qwen/Qwen2.5-0.5B/last_embed_vs_concepts.jpg)Figure G\.3:Probe accuracies across layers for all concepts inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/Qwen/Qwen2.5-1.5B/last_embed_vs_concepts.jpg)Figure G\.4:Probe accuracies across layers for all concepts inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/Qwen/Qwen2.5-3B/last_embed_vs_concepts.jpg)Figure G\.5:Probe accuracies across layers for all concepts inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/Qwen/Qwen2.5-7B/last_embed_vs_concepts.jpg)Figure G\.6:Probe accuracies across layers for all concepts inQwen2\.5\-7B
### G\.2Extended Results for Inference of Ambition

#### G\.2\.1Probing for Ambition using Nth Embedding vs\. Average Embedding

Figures[G\.7](https://arxiv.org/html/2605.28823#A7.F7)–[G\.12](https://arxiv.org/html/2605.28823#A7.F12)illustrate theAmbitionprobe accuracies across layers of all LLMs using both theNt​hN^\{th\}embedding and the average of all embeddings in the respective layer\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/google/gemma-2-2b/last_vs_avg_embed.jpg)Figure G\.7:Ambitionprobe accuracy forGemma\-2\-2Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/google/gemma-2-9b/last_vs_avg_embed.jpg)Figure G\.8:Ambitionprobe accuracy forGemma\-2\-9Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-0.5B/last_vs_avg_embed.jpg)Figure G\.9:Ambitionprobe accuracy forQwen2\.5\-0\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-1.5B/last_vs_avg_embed.jpg)Figure G\.10:Ambitionprobe accuracy forQwen2\.5\-1\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-3B/last_vs_avg_embed.jpg)Figure G\.11:Ambitionprobe accuracy forQwen2\.5\-3Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-7B/last_vs_avg_embed.jpg)Figure G\.12:Ambitionprobe accuracy forQwen2\.5\-7Busing average andNt​hN^\{th\}embeddings vs\. layer
#### G\.2\.2Ambition Probe Cross\-Check

Figures[G\.13](https://arxiv.org/html/2605.28823#A7.F13)and[G\.14](https://arxiv.org/html/2605.28823#A7.F14)show theAmbitionprobe accuracy versus probe size forGemma\-2\-2BandQwen2\.5\-0\.5B, respectively\. Figures[G\.15](https://arxiv.org/html/2605.28823#A7.F15)–[G\.21](https://arxiv.org/html/2605.28823#A7.F21)show the probe accuracies across layers for all LLMs when the probes are trained on the control task \(randomizing embeddings or labels\)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/google/gemma-2-2b/LF_sweep_last.jpg)Figure G\.13:Ambitionprobe accuracy forGemma\-2\-2Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-0.5B/LF_sweep_last.jpg)Figure G\.14:Ambitionprobe accuracy forQwen2\.5\-0\.5Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/meta-llama/Meta-Llama-3-8B/labels_vs_tokens_control.jpg)Figure G\.15:Ambitionprobe accuracy across layers inLlama\-3\-8Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/google/gemma-2-2b/labels_vs_tokens_control.jpg)Figure G\.16:Ambitionprobe accuracy across layers inGemma\-2\-2Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/google/gemma-2-9b/labels_vs_tokens_control.jpg)Figure G\.17:Ambitionprobe accuracy across layers inGemma\-2\-9Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-0.5B/labels_vs_tokens_control.jpg)Figure G\.18:Ambitionprobe accuracy across layers inQwen2\.5\-0\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-1.5B/labels_vs_tokens_control.jpg)Figure G\.19:Ambitionprobe accuracy across layers inQwen2\.5\-1\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-3B/labels_vs_tokens_control.jpg)Figure G\.20:Ambitionprobe accuracy across layers inQwen2\.5\-3Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/ambition/Qwen/Qwen2.5-7B/labels_vs_tokens_control.jpg)Figure G\.21:Ambitionprobe accuracy across layers inQwen2\.5\-7Busing random embeddings or random labels during probe training

### G\.3Extended Results for Inference of Investigation

#### G\.3\.1Probing for Investigation using Nth Embedding vs\. Average Embedding

Figures[G\.22](https://arxiv.org/html/2605.28823#A7.F22)–[G\.28](https://arxiv.org/html/2605.28823#A7.F28)illustrate theInvestigationprobe accuracies across layers of all LLMs using both theNt​hN^\{th\}embedding and the average of all embeddings in the respective layer\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/meta-llama/Meta-Llama-3-8B/last_vs_avg_embed.jpg)Figure G\.22:Investigationprobe accuracy forLlama\-3\-8Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/google/gemma-2-2b/last_vs_avg_embed.jpg)Figure G\.23:Investigationprobe accuracy forGemma\-2\-2Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/google/gemma-2-9b/last_vs_avg_embed.jpg)Figure G\.24:Investigationprobe accuracy forGemma\-2\-9Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-0.5B/last_vs_avg_embed.jpg)Figure G\.25:Investigationprobe accuracy forQwen2\.5\-0\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-1.5B/last_vs_avg_embed.jpg)Figure G\.26:Investigationprobe accuracy forQwen2\.5\-1\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-3B/last_vs_avg_embed.jpg)Figure G\.27:Investigationprobe accuracy forQwen2\.5\-3Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-7B/last_vs_avg_embed.jpg)Figure G\.28:Investigationprobe accuracy forQwen2\.5\-7Busing average andNt​hN^\{th\}embeddings vs\. layer
#### G\.3\.2Investigation Probe Cross\-Check

Figures[G\.29](https://arxiv.org/html/2605.28823#A7.F29),[G\.30](https://arxiv.org/html/2605.28823#A7.F30), and[G\.31](https://arxiv.org/html/2605.28823#A7.F31)show theInvestigationprobe accuracy versus probe size forLlama\-3\-8B,Gemma\-2\-2B, andQwen2\.5\-0\.5B, respectively, and Table[G\.1](https://arxiv.org/html/2605.28823#A7.T1)shows a summary of these results\. Figures[G\.32](https://arxiv.org/html/2605.28823#A7.F32)–[G\.38](https://arxiv.org/html/2605.28823#A7.F38)show the probe accuracies across layers for all LLMs when the probes are trained on the control task \(randomizing embeddings or labels\)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/meta-llama/Meta-Llama-3-8B/LF_sweep_last.jpg)Figure G\.29:Investigationprobe accuracy forLlama\-3\-8Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/google/gemma-2-2b/LF_sweep_last.jpg)Figure G\.30:Investigationprobe accuracy forGemma\-2\-2Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-0.5B/LF_sweep_last.jpg)Figure G\.31:Investigationprobe accuracy forQwen2\.5\-0\.5Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/meta-llama/Meta-Llama-3-8B/labels_vs_tokens_control.jpg)Figure G\.32:Investigationprobe accuracy across layers inLlama\-3\-8Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/google/gemma-2-2b/labels_vs_tokens_control.jpg)Figure G\.33:Investigationprobe accuracy across layers inGemma\-2\-2Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/google/gemma-2-9b/labels_vs_tokens_control.jpg)Figure G\.34:Investigationprobe accuracy across layers inGemma\-2\-9Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-0.5B/labels_vs_tokens_control.jpg)Figure G\.35:Investigationprobe accuracy across layers inQwen2\.5\-0\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-1.5B/labels_vs_tokens_control.jpg)Figure G\.36:Investigationprobe accuracy across layers inQwen2\.5\-1\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-3B/labels_vs_tokens_control.jpg)Figure G\.37:Investigationprobe accuracy across layers inQwen2\.5\-3Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/investigation/Qwen/Qwen2.5-7B/labels_vs_tokens_control.jpg)Figure G\.38:Investigationprobe accuracy across layers inQwen2\.5\-7Busing random embeddings or random labels during probe training- ∙\\bulletAll results are in percentage \(%\)\.
- ∙\\bullet“max” denotes 4,096 forLlama\-3\-8B, 2,304 forGemma\-2\-2B, and 896 forQwen2\.5\-0\.5B\.
- ∙\\bulletstandard deviation for each result≤\\leq2%\.

Table G\.1:Investigationprobe accuracy across model families, sizes, layers, and probe sizes

### G\.4Extended Results for Inference of Democracy

#### G\.4\.1Probing for Democracy using Nth Embedding vs\. Average Embedding

Figures[G\.39](https://arxiv.org/html/2605.28823#A7.F39)–[G\.45](https://arxiv.org/html/2605.28823#A7.F45)illustrate theDemocracyprobe accuracies across layers of all LLMs using both theNt​hN^\{th\}embedding and the average of all embeddings in the respective layer\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/meta-llama/Meta-Llama-3-8B/last_vs_avg_embed.jpg)Figure G\.39:Democracyprobe accuracy forLlama\-3\-8Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/google/gemma-2-2b/last_vs_avg_embed.jpg)Figure G\.40:Democracyprobe accuracy forGemma\-2\-2Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/google/gemma-2-9b/last_vs_avg_embed.jpg)Figure G\.41:Democracyprobe accuracy forGemma\-2\-9Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-0.5B/last_vs_avg_embed.jpg)Figure G\.42:Democracyprobe accuracy forQwen2\.5\-0\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-1.5B/last_vs_avg_embed.jpg)Figure G\.43:Democracyprobe accuracy forQwen2\.5\-1\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-3B/last_vs_avg_embed.jpg)Figure G\.44:Democracyprobe accuracy forQwen2\.5\-3Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-7B/last_vs_avg_embed.jpg)Figure G\.45:Democracyprobe accuracy forQwen2\.5\-7Busing average andNt​hN^\{th\}embeddings vs\. layer
#### G\.4\.2Democracy Probe Cross\-Check

Figures[G\.46](https://arxiv.org/html/2605.28823#A7.F46),[G\.47](https://arxiv.org/html/2605.28823#A7.F47), and[G\.48](https://arxiv.org/html/2605.28823#A7.F48)show theDemocracyprobe accuracy versus probe size forLlama\-3\-8B,Gemma\-2\-2B, andQwen2\.5\-0\.5B, respectively, and Table[G\.2](https://arxiv.org/html/2605.28823#A7.T2)shows a summary of these results\. Figures[G\.49](https://arxiv.org/html/2605.28823#A7.F49)–[G\.55](https://arxiv.org/html/2605.28823#A7.F55)show the probe accuracies across layers for all LLMs when the probes are trained on the control task \(randomizing embeddings or labels\)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/meta-llama/Meta-Llama-3-8B/LF_sweep_last.jpg)Figure G\.46:Democracyprobe accuracy forLlama\-3\-8Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/google/gemma-2-2b/LF_sweep_last.jpg)Figure G\.47:Democracyprobe accuracy forGemma\-2\-2Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-0.5B/LF_sweep_last.jpg)Figure G\.48:Democracyprobe accuracy forQwen2\.5\-0\.5Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/meta-llama/Meta-Llama-3-8B/labels_vs_tokens_control.jpg)Figure G\.49:Democracyprobe accuracy across layers inLlama\-3\-8Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/google/gemma-2-2b/labels_vs_tokens_control.jpg)Figure G\.50:Democracyprobe accuracy across layers inGemma\-2\-2Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/google/gemma-2-9b/labels_vs_tokens_control.jpg)Figure G\.51:Democracyprobe accuracy across layers inGemma\-2\-9Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-0.5B/labels_vs_tokens_control.jpg)Figure G\.52:Democracyprobe accuracy across layers inQwen2\.5\-0\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-1.5B/labels_vs_tokens_control.jpg)Figure G\.53:Democracyprobe accuracy across layers inQwen2\.5\-1\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-3B/labels_vs_tokens_control.jpg)Figure G\.54:Democracyprobe accuracy across layers inQwen2\.5\-3Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/democracy/Qwen/Qwen2.5-7B/labels_vs_tokens_control.jpg)Figure G\.55:Democracyprobe accuracy across layers inQwen2\.5\-7Busing random embeddings or random labels during probe training- ∙\\bulletAll results are in percentage \(%\)\.
- ∙\\bullet“max” denotes 4,096 forLlama\-3\-8B, 2,304 forGemma\-2\-2B, and 896 forQwen2\.5\-0\.5B\.
- ∙\\bulletstandard deviation for each result≤\\leq3%\.

Table G\.2:Democracyprobe accuracy across model families, sizes, layers, and probe sizes

### G\.5Extended Results for Inference of Envy

#### G\.5\.1Probing for Envy using Nth Embedding vs\. Average Embedding

Figures[G\.56](https://arxiv.org/html/2605.28823#A7.F56)–[G\.62](https://arxiv.org/html/2605.28823#A7.F62)illustrate theEnvyprobe accuracies across layers of all LLMs using both theNt​hN^\{th\}embedding and the average of all embeddings in the respective layer\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/meta-llama/Meta-Llama-3-8B/last_vs_avg_embed.jpg)Figure G\.56:Envyprobe accuracy forLlama\-3\-8Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/google/gemma-2-2b/last_vs_avg_embed.jpg)Figure G\.57:Envyprobe accuracy forGemma\-2\-2Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/google/gemma-2-9b/last_vs_avg_embed.jpg)Figure G\.58:Envyprobe accuracy forGemma\-2\-9Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-0.5B/last_vs_avg_embed.jpg)Figure G\.59:Envyprobe accuracy forQwen2\.5\-0\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-1.5B/last_vs_avg_embed.jpg)Figure G\.60:Envyprobe accuracy forQwen2\.5\-1\.5Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-3B/last_vs_avg_embed.jpg)Figure G\.61:Envyprobe accuracy forQwen2\.5\-3Busing average andNt​hN^\{th\}embeddings vs\. layer![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-7B/last_vs_avg_embed.jpg)Figure G\.62:Envyprobe accuracy forQwen2\.5\-7Busing average andNt​hN^\{th\}embeddings vs\. layer
#### G\.5\.2Envy Probe Cross\-Check

Figures[G\.63](https://arxiv.org/html/2605.28823#A7.F63),[G\.64](https://arxiv.org/html/2605.28823#A7.F64), and[G\.65](https://arxiv.org/html/2605.28823#A7.F65)show theEnvyprobe accuracy versus probe size forLlama\-3\-8B,Gemma\-2\-2B, andQwen2\.5\-0\.5B, respectively, and Table[G\.3](https://arxiv.org/html/2605.28823#A7.T3)shows a summary of these results\. Figures[G\.66](https://arxiv.org/html/2605.28823#A7.F66)–[G\.72](https://arxiv.org/html/2605.28823#A7.F72)show the probe accuracies across layers for all LLMs when the probes are trained on the control task \(randomizing embeddings or labels\)\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/meta-llama/Meta-Llama-3-8B/LF_sweep_last.jpg)Figure G\.63:Envyprobe accuracy forLlama\-3\-8Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/google/gemma-2-2b/LF_sweep_last.jpg)Figure G\.64:Envyprobe accuracy forGemma\-2\-2Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-0.5B/LF_sweep_last.jpg)Figure G\.65:Envyprobe accuracy forQwen2\.5\-0\.5Bas a function of probe size![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/meta-llama/Meta-Llama-3-8B/labels_vs_tokens_control.jpg)Figure G\.66:Envyprobe accuracy across layers inLlama\-3\-8Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/google/gemma-2-2b/labels_vs_tokens_control.jpg)Figure G\.67:Envyprobe accuracy across layers inGemma\-2\-2Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/google/gemma-2-9b/labels_vs_tokens_control.jpg)Figure G\.68:Envyprobe accuracy across layers inGemma\-2\-9Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-0.5B/labels_vs_tokens_control.jpg)Figure G\.69:Envyprobe accuracy across layers inQwen2\.5\-0\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-1.5B/labels_vs_tokens_control.jpg)Figure G\.70:Envyprobe accuracy across layers inQwen2\.5\-1\.5Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-3B/labels_vs_tokens_control.jpg)Figure G\.71:Envyprobe accuracy across layers inQwen2\.5\-3Busing random embeddings or random labels during probe training![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp1/envy/Qwen/Qwen2.5-7B/labels_vs_tokens_control.jpg)Figure G\.72:Envyprobe accuracy across layers inQwen2\.5\-7Busing random embeddings or random labels during probe training- ∙\\bulletAll results are in percentage \(%\)\.
- ∙\\bullet“max” denotes 4,096 forLlama\-3\-8B, 2,304 forGemma\-2\-2B, and 896 forQwen2\.5\-0\.5B\.
- ∙\\bulletstandard deviation for each result≤\\leq1%\.

Table G\.3:Envyprobe accuracy across model families, sizes, layers, and probe sizes

## Appendix HExtended Results for Waxing and Waning of Concepts

In addition to investigating whether concepts wax and wane in an LLM’s embeddings as its context expands, we also studied how this behavior varies across layers to identify which layer best captures this change\. For each layer, we generated kernel density estimation \(KDE\) plots showing the distribution of the probe’s output values for the target concept across different story segments \(e\.g\., paragraph 1, transition sentence 1, paragraph 2\), aggregated over all stories in the concept dataset\. Figure[H\.1](https://arxiv.org/html/2605.28823#A8.F1)shows an example for theambitionprobe at layer 13 ofLlama\-3\-8B\(using the final subword token embedding\), with KDEs for the transition sentences shown in shades of green and those for paragraphs in shades of red\.

A layer that accurately captures waxing and waning should show the transition KDEs concentrated above 0\.5 and paragraph KDEs below 0\.5\. In contrast, a layer that does not capture the waxing and waning would either have all KDEs for paragraphs and transitions concentrated around the same point, as shown in Figure[H\.2](https://arxiv.org/html/2605.28823#A8.F2)\(using cumulative mean embeddings\), or have the KDEs for each segment widely distributed\.

We stacked these plots for all LLM layers to identify the one that best tracks the waxing and waning, as shown ahead\. Table[H\.1](https://arxiv.org/html/2605.28823#A8.T1)shows the layer in each LLM that best captures the waxing and waning for each investigated concept\.

We use the probes for the corresponding layers to track the model embedding’s word\-level behavior\. Using the story datasets, we obtained an aggregate view of each probe’s outputs\. This was computed as the average probe output for each word position across all 50 stories\. For example, the aggregate value for word number 3 in sentence 15 is determined by the average of the corresponding probe outputs for all the words in that position across all stories\. To obtain correct average values, all sentences at the same sentence position across stories must have the same number of words, which might not be the case\. To alleviate this issue, we applied left\-padding to each sentence to align its length with longest sentence at that position\. The padded tokens were excluded from the averaging computation\.

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/meta-llama/Meta-Llama-3-8B/last/density_plot_layer13.jpg)Figure H\.1:Kernel density estimation of theAmbitionprobe’s output values across story segments, aggregated over all stories, from layer 13 ofLlama\-3\-8B\(using final subword token embeddings\)![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/meta-llama/Meta-Llama-3-8B/average/density_plot_layer13.jpg)Figure H\.2:Kernel density estimation of theAmbitionprobe’s output values across story segments, aggregated over all stories, from layer 13 ofLlama\-3\-8B\(using cumulative mean embeddings\)Table H\.1:LLM layers that best capture the waxing and waning of the investigated concepts### H\.1Tracking Waxing and Waning of Ambition

#### H\.1\.1Layer\-Wise KDEs for Ambition Probe Outputs

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/meta-llama/Meta-Llama-3-8B/last/density_stacked.jpg)Figure H\.3:Layer\-wise KDEs forambitionprobe outputs inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/google/gemma-2-2b/last/density_stacked.jpg)Figure H\.4:Layer\-wise KDEs forambitionprobe outputs inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/google/gemma-2-9b/last/density_stacked.jpg)Figure H\.5:Layer\-wise KDEs forambitionprobe outputs inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-0.5B/last/density_stacked.jpg)Figure H\.6:Layer\-wise KDEs forambitionprobe outputs inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-1.5B/last/density_stacked.jpg)Figure H\.7:Layer\-wise KDEs forambitionprobe outputs inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-3B/last/density_stacked.jpg)Figure H\.8:Layer\-wise KDEs forambitionprobe outputs inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-7B/last/density_stacked.jpg)Figure H\.9:Layer\-wise KDEs forambitionprobe outputs inQwen2\.5\-7B
#### H\.1\.2Ambition Probe Results for Best Layers

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/google/gemma-2-2b/average/prominence_aggregate.jpg)Figure H\.10:Ambitionprobe outputs across words using both representative embeddings inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/google/gemma-2-9b/average/prominence_aggregate.jpg)Figure H\.11:Ambitionprobe outputs across words using both representative embeddings inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-0.5B/average/prominence_aggregate.jpg)Figure H\.12:Ambitionprobe outputs across words using both representative embeddings inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-1.5B/average/prominence_aggregate.jpg)Figure H\.13:Ambitionprobe outputs across words using both representative embeddings inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-3B/average/prominence_aggregate.jpg)Figure H\.14:Ambitionprobe outputs across words using both representative embeddings inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/ambition/Qwen/Qwen2.5-7B/average/prominence_aggregate.jpg)Figure H\.15:Ambitionprobe outputs across words using both representative embeddings inQwen2\.5\-7B

### H\.2Tracking Waxing and Waning of Investigation

#### H\.2\.1Layer\-Wise KDEs for Investigation Probe Outputs

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/meta-llama/Meta-Llama-3-8B/last/density_stacked.jpg)Figure H\.16:Layer\-wise KDEs forinvestigationprobe outputs inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/google/gemma-2-2b/last/density_stacked.jpg)Figure H\.17:Layer\-wise KDEs forinvestigationprobe outputs inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/google/gemma-2-9b/last/density_stacked.jpg)Figure H\.18:Layer\-wise KDEs forinvestigationprobe outputs inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-0.5B/last/density_stacked.jpg)Figure H\.19:Layer\-wise KDEs forinvestigationprobe outputs inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-1.5B/last/density_stacked.jpg)Figure H\.20:Layer\-wise KDEs forinvestigationprobe outputs inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-3B/last/density_stacked.jpg)Figure H\.21:Layer\-wise KDEs forinvestigationprobe outputs inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-7B/last/density_stacked.jpg)Figure H\.22:Layer\-wise KDEs forinvestigationprobe outputs inQwen2\.5\-7B
#### H\.2\.2Investigation Probe Results for Best Layers

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/meta-llama/Meta-Llama-3-8B/average/prominence_aggregate.jpg)Figure H\.23:Investigationprobe outputs across words using both representative embeddings inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/google/gemma-2-2b/average/prominence_aggregate.jpg)Figure H\.24:Investigationprobe outputs across words using both representative embeddings inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/google/gemma-2-9b/average/prominence_aggregate.jpg)Figure H\.25:Investigationprobe outputs across words using both representative embeddings inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-0.5B/average/prominence_aggregate.jpg)Figure H\.26:Investigationprobe outputs across words using both representative embeddings inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-1.5B/average/prominence_aggregate.jpg)Figure H\.27:Investigationprobe outputs across words using both representative embeddings inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-3B/average/prominence_aggregate.jpg)Figure H\.28:Investigationprobe outputs across words using both representative embeddings inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/investigation/Qwen/Qwen2.5-7B/average/prominence_aggregate.jpg)Figure H\.29:Investigationprobe outputs across words using both representative embeddings inQwen2\.5\-7B

### H\.3Tracking Waxing and Waning of Democracy

#### H\.3\.1Layer\-Wise KDEs for Democracy Probe Outputs

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/meta-llama/Meta-Llama-3-8B/last/density_stacked.jpg)Figure H\.30:Layer\-wise KDEs fordemocracyprobe outputs inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/google/gemma-2-2b/last/density_stacked.jpg)Figure H\.31:Layer\-wise KDEs fordemocracyprobe outputs inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/google/gemma-2-9b/last/density_stacked.jpg)Figure H\.32:Layer\-wise KDEs fordemocracyprobe outputs inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-0.5B/last/density_stacked.jpg)Figure H\.33:Layer\-wise KDEs fordemocracyprobe outputs inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-1.5B/last/density_stacked.jpg)Figure H\.34:Layer\-wise KDEs fordemocracyprobe outputs inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-3B/last/density_stacked.jpg)Figure H\.35:Layer\-wise KDEs fordemocracyprobe outputs inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-7B/last/density_stacked.jpg)Figure H\.36:Layer\-wise KDEs fordemocracyprobe outputs inQwen2\.5\-7B
#### H\.3\.2Democracy Probe Results for Best Layers

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/meta-llama/Meta-Llama-3-8B/average/prominence_aggregate.jpg)Figure H\.37:Democracyprobe outputs across words using both representative embeddings inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/google/gemma-2-2b/average/prominence_aggregate.jpg)Figure H\.38:Democracyprobe outputs across words using both representative embeddings inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/google/gemma-2-9b/average/prominence_aggregate.jpg)Figure H\.39:Democracyprobe outputs across words using both representative embeddings inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-0.5B/average/prominence_aggregate.jpg)Figure H\.40:Democracyprobe outputs across words using both representative embeddings inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-1.5B/average/prominence_aggregate.jpg)Figure H\.41:Democracyprobe outputs across words using both representative embeddings inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-3B/average/prominence_aggregate.jpg)Figure H\.42:Democracyprobe outputs across words using both representative embeddings inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/democracy/Qwen/Qwen2.5-7B/average/prominence_aggregate.jpg)Figure H\.43:Democracyprobe outputs across words using both representative embeddings inQwen2\.5\-7B

### H\.4Tracking Waxing and Waning of Envy

#### H\.4\.1Layer\-Wise KDEs for Envy Probe Outputs

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/meta-llama/Meta-Llama-3-8B/last/density_stacked.jpg)Figure H\.44:Layer\-wise KDEs forenvyprobe outputs inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/google/gemma-2-2b/last/density_stacked.jpg)Figure H\.45:Layer\-wise KDEs forenvyprobe outputs inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/google/gemma-2-9b/last/density_stacked.jpg)Figure H\.46:Layer\-wise KDEs forenvyprobe outputs inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-0.5B/last/density_stacked.jpg)Figure H\.47:Layer\-wise KDEs forenvyprobe outputs inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-1.5B/last/density_stacked.jpg)Figure H\.48:Layer\-wise KDEs forenvyprobe outputs inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-3B/last/density_stacked.jpg)Figure H\.49:Layer\-wise KDEs forenvyprobe outputs inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-7B/last/density_stacked.jpg)Figure H\.50:Layer\-wise KDEs forenvyprobe outputs inQwen2\.5\-7B
#### H\.4\.2Envy Probe Results for Best Layers

![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/meta-llama/Meta-Llama-3-8B/average/prominence_aggregate.jpg)Figure H\.51:Envyprobe outputs across words using both representative embeddings inLlama\-3\-8B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/google/gemma-2-2b/average/prominence_aggregate.jpg)Figure H\.52:Envyprobe outputs across words using both representative embeddings inGemma\-2\-2B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/google/gemma-2-9b/average/prominence_aggregate.jpg)Figure H\.53:Envyprobe outputs across words using both representative embeddings inGemma\-2\-9B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-0.5B/average/prominence_aggregate.jpg)Figure H\.54:Envyprobe outputs across words using both representative embeddings inQwen2\.5\-0\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-1.5B/average/prominence_aggregate.jpg)Figure H\.55:Envyprobe outputs across words using both representative embeddings inQwen2\.5\-1\.5B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-3B/average/prominence_aggregate.jpg)Figure H\.56:Envyprobe outputs across words using both representative embeddings inQwen2\.5\-3B![Refer to caption](https://arxiv.org/html/2605.28823v1/figures/exp2/envy/Qwen/Qwen2.5-7B/average/prominence_aggregate.jpg)Figure H\.57:Envyprobe outputs across words using both representative embeddings inQwen2\.5\-7B

## Appendix IDataset Overview

We release all the datasets created and used in this paper to be used by others for concept exploration in LLMs\. Table[I\.1](https://arxiv.org/html/2605.28823#A9.T1)on the next page describes the contents of each file\.

Table I\.1:Description of released files for the datasets