Embedding-Based Educational Taxonomies

# Embedding-Based Educational Taxonomies
Source: [https://arxiv.org/html/2610.33694](https://arxiv.org/html/2610.33694)
Kaveh Karbasi††thanks:Tempus AI, Inc\. had no role in the planning, execution, funding, or review of this work\.Affiliation:Tempus AI, Inc\.Gregory PottieAffiliation:University of California, Los Angeles

###### Abstract

Educational taxonomies organize educational content, such as exercise problems, into concepts, which are also called skills or knowledge components\. These taxonomies define learning objectives in educational applications such as intelligent tutoring systems\. Developed through human expertise and iterative refinement, they also provide fundamental structure for curricula, assessment, and models of student learning\. Recent advances in semantic text embeddings make it possible to construct alternative educational taxonomies computationally from the content itself\. In this paper, we investigate this possibility by clustering embeddings of math questions to construct embedding\-based educational taxonomies\. We compare these taxonomies with the existing Common Core State Standards \(CCSS\)\-based knowledge\-component taxonomies in two ASSISTments datasets and evaluate their utility in Deep Knowledge Tracing, a downstream task that relies on taxonomic mappings of questions to knowledge components\. The embedding\-based taxonomies exhibit substantial but incomplete correspondence with the existing taxonomies, revealing both preserved and alternative organizations of educational content\. When used for knowledge tracing, they improve predictive performance by approximately 3–5 AUC percentage points\. These results demonstrate the potential of semantic embeddings for automatically constructing educational taxonomies that both recover structure present in human\-developed taxonomies and provide useful alternative representations for downstream educational modeling\.

## 1Introduction

Educational taxonomies organize educational content, such as exercise problems, into concepts\. In educational data and modeling, these concepts are often called skills or knowledge components \(KCs\)\. A taxonomy specifies which questions are treated as instances of related knowledge, giving educators and computer systems a common way to describe content and learning objectives\. These mappings provide structure for curricula, assessment, educational applications, and models of student learning[Koedinger et al\. \(2012\)](https://arxiv.org/html/2610.33694#bib.bib11)\.

Educational taxonomies are generally developed through human expertise and iterative refinement\. The Common Core State Standards, for example, were developed through research, educator and expert input, comparison with existing standards, and public feedback[Common Core State Standards Initiative \(n\.d\.\)](https://arxiv.org/html/2610.33694#bib.bib7)\. Such systems encode decisions about which distinctions matter for curricula and assessment, while KC models use those distinctions to relate educational tasks, instruction, and student learning[Porter et al\. \(2011\)](https://arxiv.org/html/2610.33694#bib.bib20);[Koedinger et al\. \(2012\)](https://arxiv.org/html/2610.33694#bib.bib11)\. They are therefore useful educational frameworks, not direct evidence that knowledge is organized in one uniquely correct way\.

These taxonomies also provide the categories that educational technologies use to organize content and model student learning\. Intelligent tutoring systems connect educational materials to the concepts they are intended to teach or assess[Koedinger et al\. \(2012\)](https://arxiv.org/html/2610.33694#bib.bib11);[Feng et al\. \(2009\)](https://arxiv.org/html/2610.33694#bib.bib9)\. Knowledge tracing uses sequences of questions and responses, along with the concepts assigned to those questions, to estimate student knowledge and predict later performance[Corbett and Anderson \(1995\)](https://arxiv.org/html/2610.33694#bib.bib8);[Abdelrahman et al\. \(2023\)](https://arxiv.org/html/2610.33694#bib.bib2)\. Deep Knowledge Tracing \(DKT\) represents each response with its assigned KC, so earlier performance on a KC can inform later predictions[Piech et al\. \(2015\)](https://arxiv.org/html/2610.33694#bib.bib19)\. Systems that recommend exercises similarly use concept labels and estimates of student knowledge to select individualized practice[Ai et al\. \(2019\)](https://arxiv.org/html/2610.33694#bib.bib3)\.

Recent advances in text embeddings make it possible to organize educational content using patterns in the content itself\. Embeddings represent question text in a numerical space where similarities in language and meaning can be compared[Reimers and Gurevych \(2019\)](https://arxiv.org/html/2610.33694#bib.bib22)\. Clustering these representations groups nearby questions and assigns each question to one category[Matsuda et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib15)\.

In this paper, we use semantic embeddings of math question text to induce embedding\-based taxonomies\. Here, a taxonomy refers to a mapping between content or questions and content categories or knowledge components\. An induced taxonomy is one constructed from text embeddings\. We compare these induced taxonomies with the original expert\-developed taxonomies and test their utility in a downstream educational task that relies on question\-to\-KC mappings\.

We study questions with available text from ASSISTments 2012 and 2017, two releases from a widely used benchmark family in knowledge tracing[Feng et al\. \(2009\)](https://arxiv.org/html/2610.33694#bib.bib9);[Prihar et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib21);[Abdelrahman et al\. \(2023\)](https://arxiv.org/html/2610.33694#bib.bib2)\. We embed the question text, reduce the representations with principal component analysis \(PCA\), and cluster them withkk\-means to construct taxonomies from embeddings\. We first compare each taxonomy built from embeddings with the original KC taxonomy supplied with the dataset, using several measures of how the questions are grouped, along with category counts\. We then replace the original mapping from questions to KCs with the induced mapping and compare DKT performance using the area under the receiver operating characteristic curve \(AUC\)\. Within each dataset, we use the same student\-response records, sequence order, student split, DKT model family, training procedure, Bayesian hyperparameter search, and trial budget\. Because an induced taxonomy may use a different number of categories, the number of KC labels and the number of model parameters tied to those labels may also differ\. We tune DKT separately for each taxonomy, so the selected DKT hyperparameters may differ\. The code and supplementary materials are available in a public GitHub repository\.111[https://github\.com/DKT\-PROJECTS\-UCLA/KC\_Clusters](https://github.com/DKT-PROJECTS-UCLA/KC_Clusters)

We ask three research questions:

RQ1:How closely do the induced and original KC taxonomies correspond?

RQ2:How does correspondence change as the induced taxonomy uses more categories?

RQ3:Under which taxonomy does DKT achieve higher AUC: the original KC taxonomy or the embedding\-induced taxonomy?

## 2Related Work

### 2\.1Educational Taxonomies and Knowledge Components

KC models connect performance on related tasks to proposed units of knowledge used in teaching and models of student learning[Koedinger et al\. \(2012\)](https://arxiv.org/html/2610.33694#bib.bib11)\. A mapping from questions to KCs therefore represents a hypothesis about the knowledge required by each question\. These models are designed for educational use; they are not direct measurements of how students think\. Learning Factors Analysis shows that student data can be used to evaluate and refine an expert KC model[Cen et al\. \(2006\)](https://arxiv.org/html/2610.33694#bib.bib6)\. Whereas that approach compares possible refinements using student performance, we construct alternative categories from question content\. We therefore use the existing KC taxonomy as a valuable reference, not as the only possible way to group the questions\. Expert\-developed taxonomies reflect instructional organization; induced taxonomies offer complementary groupings derived from question text\. Whether these groupings align with how students organize or learn that content remains an open question\.

### 2\.2Embedding\-Based Clustering and Automated KC Discovery

Sentence embeddings represent text in a space that supports semantic comparison and clustering[Reimers and Gurevych \(2019\)](https://arxiv.org/html/2610.33694#bib.bib22)\. Text similarity can suggest shared content, but it does not by itself establish a common solution method or role in teaching\. SMART useskk\-means to cluster representations of instructional and assessment text in which terms are weighted by their frequency within a document and rarity across documents \(TF–IDF\), compares the induced categories with expert categories, and evaluates the resulting mapping with an Additive Factors Model \(AFM\)[Matsuda et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib15)\. KCluster uses language\-model\-based question similarity and affinity propagation to derive KCs, compares them with an expert taxonomy, and also evaluates them with AFM[Wei et al\. \(2025\)](https://arxiv.org/html/2610.33694#bib.bib28)\. These are the clustering studies most closely related to our work, although neither uses ASSISTments or DKT\. Moon et al\. extract KCs from question text and images, group them using embeddings, and evaluate the resulting categories with several knowledge\-tracing models, including DKT[Moon et al\. \(2025\)](https://arxiv.org/html/2610.33694#bib.bib17)\.

Together, these studies show that educational categories can be constructed from course and question content\. Our study builds on this work by comparing induced and original taxonomies across every tested combination of PCA dimension and number of induced categories \(KK\), then evaluating the induced mappings in DKT\. We also ask whether settings where the taxonomies match more closely tend to produce higher AUC\. The core methodology was developed independently of SMART and KCluster, which we identified as closely related approaches during subsequent literature review\.

### 2\.3Knowledge Tracing and Text\-Aware Student Modeling

Knowledge tracing uses question–response histories to model how student knowledge develops over time[Corbett and Anderson \(1995\)](https://arxiv.org/html/2610.33694#bib.bib8);[Abdelrahman et al\. \(2023\)](https://arxiv.org/html/2610.33694#bib.bib2)\. DKT uses a recurrent neural network to predict later responses from earlier responses and their KC labels[Piech et al\. \(2015\)](https://arxiv.org/html/2610.33694#bib.bib19)\. For example, if two questions share a KC label, DKT can use a student’s earlier performance on that KC when predicting a later response; under our method, the same process uses the induced KC labels instead\. Methods that use question text add question representations to student models while retaining the existing skill labels[Su et al\. \(2018\)](https://arxiv.org/html/2610.33694#bib.bib25);[Liu et al\. \(2021\)](https://arxiv.org/html/2610.33694#bib.bib12)\. Our approach uses embeddings before DKT is trained: the embeddings define an alternative mapping from questions to categories, and DKT is then trained with the induced labels\. This lets us evaluate the mapping in two ways: how closely it matches the original taxonomy and the test AUC achieved by DKT with the induced labels\.

## 3Method

Figure[1](https://arxiv.org/html/2610.33694#S3.F1)illustrates knowledge tracing with an induced taxonomy and the corresponding relabeling used for DKT evaluation\. We first derive a mapping from questions to categories that has one level, using question text alone\. We then replace the KC mapping supplied with the dataset with the induced mapping in DKT while keeping the records of student answers and their order unchanged\.

![Refer to caption](https://arxiv.org/html/2610.33694v1/method_kc_relabeling.png)Figure 1:Knowledge tracing with an induced taxonomy and sequence relabeling\. Panel \(a\) conceptually illustrates DKT tracking induced KCs; the illustrated mastery states are conceptual and do not represent measured student knowledge\. Panel \(b\) shows the actual relabeling procedure for DKT evaluation: the original KC label in each interaction is replaced with the induced label while questions, responses, sequence order, student splits, and evaluation remain unchanged\.### 3\.1Data and Building the Taxonomies

We use ASSISTments 2012 and 2017, two releases from a widely used benchmark family in knowledge tracing[Feng et al\. \(2009\)](https://arxiv.org/html/2610.33694#bib.bib9);[Prihar et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib21);[Abdelrahman et al\. \(2023\)](https://arxiv.org/html/2610.33694#bib.bib2), because their student interactions can be linked to question text from the ASSISTments Problem Bodies \(PB\) resource[ASSISTments Project \(n\.d\.\)](https://arxiv.org/html/2610.33694#bib.bib4)\. The full 2012 release contains 179,999 questions and 265 KCs, of which 32,792 questions and 162 KCs belong to the PB text\-linked subset\. The full 2017 release contains 3,162 questions and 102 KCs, of which 638 questions and 68 KCs belong to the PB text\-linked subset\. These PB subsets are the text\-bearing questions and KCs used for our text\-based analyses\. The largest category in the original 2017 taxonomy, noskill, is a non\-substantive placeholder covering 142 of the 638 questions \(22%\)\. We exclude ASSISTments 2009 because its archived release is sensitive to preprocessing choices[Xiong et al\. \(2016\)](https://arxiv.org/html/2610.33694#bib.bib29)\. We refer to the dataset\-provided question\-to\-KC mapping in each release as the original taxonomy\. Table[1](https://arxiv.org/html/2610.33694#S3.T1)summarizes the dataset sizes, the category counts tested, and the PCA andKKsetting reported for each dataset in the taxonomy comparison and DKT results\.

Table 1:Dataset and taxonomy summary\. Total counts describe the full releases; PB counts describe the Problem Bodies text\-linked question/KC subsets used for text\-based analyses\. Student and response counts describe the text\-linked subset used for DKT evaluation\. For both datasets, we construct induced taxonomies using outputs from OpenAI’stext\-embedding\-3\-smallandtext\-embedding\-3\-large, shortened through the API’sdimensionsparameter to 768 and 1,536 dimensions, respectively[OpenAI \(2026\)](https://arxiv.org/html/2610.33694#bib.bib18); we use the PCA dimensions\{10,20,40,60,80,100,120,160\}\\\{10,20,40,60,80,100,120,160\\\}, withKKvalues\{50,100,150,200,250,300\}\\\{50,100,150,200,250,300\\\}for 2012 and\{20,40,60,80,100,150\}\\\{20,40,60,80,100,150\\\}for 2017\. PCA andkk\-means use a random seed of 42;kk\-means uses 10 initializations and at most 300 iterations\. The reported \(PCA,KK\) settings in Table[2](https://arxiv.org/html/2610.33694#S4.T2)are \(100, 300\) for 2012 and \(10, 100\) for 2017\. The PCA dimension andKKare fixed before each DKT Bayesian search and are not selected by it\.We convert the HTML in eachproblem\_bodyto plain text, retain image descriptions, remove scripts and style information, and normalize whitespace\. We construct each induced taxonomy from these cleaned question texts\. The original KC labels and student responses are not used during construction\. Two models embed each question, PCA reduces the vectors, andkk\-means assigns each question to the nearest cluster center\. We evaluate eight PCA dimensions and six values ofKK, the number of induced categories, for each dataset\. The tested grids \(2012: 50–300; 2017: 20–150\) span values below, near, and above the original PB KC counts of 162 and 68, respectively\. Each taxonomy assigns one label per question and has one level, not a hierarchy\. The taxonomy is constructed from the complete set of linked question texts before splitting students\. Following the standard evaluation for knowledge tracing implemented in pyKT[Liu et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib13), we split the data by student and evaluate students held out from training\. The test set therefore contains students not seen during training rather than previously unseen questions\.

Appendix[A](https://arxiv.org/html/2610.33694#A1)documents the permitted examples of question text, and Appendix[B](https://arxiv.org/html/2610.33694#A2)gives the exact embedding models, PCA dimensions, values ofKK, clustering settings, and procedure used to select DKT settings\.

Problem Bodies were accessed under the ASSISTments terms of use for the data\. The embedding inputs consisted only of cleaned text from theproblem\_bodyfield and excluded grade information, original KC labels, and records of student answers\. Because the access terms prohibit redistribution of the questions, neither the paper nor the public GitHub repository reproduces the original question text\.

### 3\.2Taxonomy Comparison

We compare questions that have both an original and an induced label using the measures summarized in Appendix Table[3](https://arxiv.org/html/2610.33694#A3.T3)\. Homogeneity measures whether each induced category primarily contains questions from one original category, while completeness measures whether each original category remains concentrated within one induced category\. The remaining measures ask how similarly the two taxonomies group questions, whether pairs of questions stay together, and how different the groupings are overall\. Because some scores change simply when the induced taxonomy uses more categories, we interpret them together rather than treating any one score as a complete evaluation\. We report induced category sizes separately\. For the reported 2017 DKT setting, Appendix Figure[5](https://arxiv.org/html/2610.33694#A3.F5)shows how the 25 largest original categories are distributed across the induced taxonomy\. Together, these measures describe how categories stay together, split, or mix across the two taxonomies\. Educational usefulness requires separate evidence\.

### 3\.3DKT Evaluation

We use DKT because it can use a student’s performance on one KC when predicting later answers[Piech et al\. \(2015\)](https://arxiv.org/html/2610.33694#bib.bib19)\. KC labels are part of each sequence given to DKT, so replacing the original labels tests how DKT performs with the induced taxonomy as a whole\. We keep every question, response, and sequence position fixed and replace the original mapping from questions to KCs with the corresponding induced mapping\. pyKT maps the resulting labels to consecutive concept identifiers[Liu et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib13)\. VaryingKKchanges the number of labels and can also change the number of model parameters tied to those labels\. Figure[1](https://arxiv.org/html/2610.33694#S3.F1)b illustrates this relabeling\.

We run a separate Bayesian search for the original taxonomy and for each induced taxonomy defined by an embedding model, PCA dimension, andKK\. Within each dataset, all searches use the same DKT model family, candidate settings, number of trials, student split, training procedure, and selection rule\. We reserve 20% of students for final testing, giving an approximate 64/16/20 training, validation, and test split\. Within each search, we select the completed run with the highest validation AUC and report its test AUC\. The embedding model, PCA dimension, andKKare fixed before the search\. Testing both embedding models across eight PCA dimensions and six values ofKKyields 96 induced taxonomies per dataset\. Table[2](https://arxiv.org/html/2610.33694#S4.T2)reports the highest test AUC across the induced taxonomies and the embedding model, PCA dimension, andKKthat produced it; Figure[4](https://arxiv.org/html/2610.33694#S4.F4)shows the grid of PCA dimensions andKKvalues for that embedding model\. Appendix[B](https://arxiv.org/html/2610.33694#A2)gives the exact search spaces and training settings\.

To test whether closer agreement between the taxonomies is related to higher AUC, we analyze the 48 settings of PCA dimension andKKin each dataset\. At each setting, we average each structural score across the two embedding models and separately average the two test AUC values\. We compute Spearman correlations, then repeat the analysis after removing rank patterns linked to PCA dimension andKK\. We apply the Benjamini–Hochberg correction separately to each set of 15 comparisons[Benjamini and Hochberg \(1995\)](https://arxiv.org/html/2610.33694#bib.bib5)\. Appendix[D](https://arxiv.org/html/2610.33694#A4)reports the full procedure and coefficients; the analysis shows associations, not causal effects\.

## 4Results

### 4\.1RQ1: Comparing the Original and Induced Taxonomies

Table[2](https://arxiv.org/html/2610.33694#S4.T2)shows much stronger agreement for ASSISTments 2012 than for ASSISTments 2017 at the two reported settings\. For 2012, homogeneity, completeness, and normalized mutual information \(NMI\) are \.930, \.717, and \.809; for 2017, they are \.681, \.509, and \.582\. Neither setting yields the highest adjusted Rand index \(ARI\) or NMI across all tested settings, showing that the settings reported for DKT are not simply those with the closest match\. The full comparison results appear in Appendix[C](https://arxiv.org/html/2610.33694#A3); Appendix Table[5](https://arxiv.org/html/2610.33694#A3.T5)gives additional comparison scores at these two settings\. Appendix Figure[5](https://arxiv.org/html/2610.33694#A3.F5)shows how the 142 questions in noskill are distributed across the induced taxonomy\.

![Refer to caption](https://arxiv.org/html/2610.33694v1/assist2012_contingency_heatmap.png)Figure 2:Distribution of questions from the 25 largest categories in the 2012 original taxonomy across the corresponding induced taxonomy\. The figure usestext\-embedding\-3\-large, PCA=100=100, andK=300K=300\. The left heatmap shows question counts and the right heatmap shows the share of each original category in each induced category\. We include the three induced categories containing the most questions from each original category and combine the rest asOther\. The identifiers for induced categories are arbitrary labels and do not indicate an order\.Figure[2](https://arxiv.org/html/2610.33694#S4.F2)shows that questions from large original 2012 categories are often distributed across several induced categories\. These divisions may reflect finer mathematical distinctions, recurring wording or templates, or the effect of using more categories\. This analysis does not show whether these divisions matter for teaching\.

Table 2:Taxonomy comparison and DKT results at the reported settings\. Scores for taxonomy agreement are averaged across the two embedding models at each listed PCA dimension andKK\. The embedding column identifies the model used for the induced taxonomy with the highest observed test AUC\. The induced value is the highest observed test AUC across the tested combinations of embedding model, PCA dimension, andKK\.Δ\\DeltaAUC is induced minus original\.
### 4\.2RQ2: Effect of the Number of Categories

Figure 3:Mean homogeneity and completeness as the induced taxonomy uses more categories\. At eachKK, the lines show averages over the eight PCA dimensions and two embedding models; bands show the full range across those 16 settings and are not confidence intervals\. In 2012, homogeneity generally increases while completeness decreases\. In 2017, both generally increase, although inverse purity and pairwise recall decline over the sameKKrange\. Appendix Figure[7](https://arxiv.org/html/2610.33694#A3.F7)gives the full results for each embedding model and PCA dimension\.Figure[3](https://arxiv.org/html/2610.33694#S4.F3)shows how homogeneity and completeness change asKKincreases\. In ASSISTments 2012, homogeneity generally rises while completeness falls: induced categories become more concentrated, while each original KC is distributed across more induced categories\. In ASSISTments 2017, both averages generally rise over the tested range, but inverse purity and pairwise recall decline, as shown in Appendix Table[6](https://arxiv.org/html/2610.33694#A3.T6)\. The measures therefore capture different effects of using more categories and do not indicate improvement in every aspect of the match\.

### 4\.3RQ3: Taxonomies and DKT Performance

Table[2](https://arxiv.org/html/2610.33694#S4.T2)reports that the highest test AUC among the induced taxonomies exceeds the AUC with the original taxonomy in both datasets\. For ASSISTments 2012, AUC rises from \.7367 with the original taxonomy to \.7841, a gain of \.0474\. For ASSISTments 2017, it rises from \.7131 to \.7403, a gain of \.0272\. Figure[4](https://arxiv.org/html/2610.33694#S4.F4)shows that, fortext\-embedding\-3\-large, test AUC exceeds the original\-taxonomy baseline at all 48 PCA/KKgrid settings in each dataset, including settings with fewer categories than the corresponding original taxonomy within the PB subset\.

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/chapter6_assist2012_improvement_heatmap.png)

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/chapter6_assist2017_improvement_heatmap.png)

Figure 4:Change in test AUC when using the induced taxonomy instead of the original taxonomy\. The figure usestext\-embedding\-3\-large, which produced the highest observed test AUC in both datasets\. Each cell shows the change in test AUC for one PCA andKKsetting\. The test AUC at every displayed setting exceeds the baseline with the original taxonomy\. We also testtext\-embedding\-3\-small, giving 96 induced taxonomies per dataset in total\.Before adjustment, many comparison measures correlate strongly with AUC because both vary with PCA dimension andKK\. After accounting for those patterns and correcting for 15 comparisons, none of the 2012 correlations remains statistically significant\. In 2017, several measures of closer agreement are associated with lower AUC, while a larger variation of information score is associated with higher AUC\. Closer agreement is therefore not consistently associated with higher DKT AUC across the tested settings\. These correlations do not show that differences from the original taxonomy cause the gains\. Appendix[D](https://arxiv.org/html/2610.33694#A4)reports the full coefficients in Table[7](https://arxiv.org/html/2610.33694#A4.T7)and Figure[8](https://arxiv.org/html/2610.33694#A4.F8)\.

## 5Discussion

The original and induced KC taxonomies organize the same questions using different evidence\. The stronger agreement in ASSISTments 2012 suggests that patterns in question text align more closely with the original KC structure there than in ASSISTments 2017\. In both datasets, however, original categories are split across several induced categories\. These splits may reflect mathematical content, recurring wording, or the use of more induced categories; the current analysis cannot determine which explanation is correct\.

The 2017 original taxonomy’s largest category,noskill, is a non\-substantive placeholder covering 142 of 638 questions \(22%\)\. Becausenoskillgroups questions by the absence of an assigned KC rather than shared content, a taxonomy constructed from question content has no reason to reproduce it as a single category\. Appendix Figure[5](https://arxiv.org/html/2610.33694#A3.F5)shows the induced taxonomy distributing these questions across several categories\. The original\-taxonomy baseline retainsnoskill, so the reported AUC difference compares DKT with this placeholder against DKT with the complete induced mapping\. This redistribution may contribute to both the lower correspondence and the higher AUC observed in 2017, although the experiment does not isolate its contribution from the other changes in the mapping\.

The DKT results address a different question: how well each taxonomy supports prediction\. The induced taxonomy with the highest observed test AUC improves test AUC by \.0474 in 2012 and \.0272 in 2017, and Figure[4](https://arxiv.org/html/2610.33694#S4.F4)shows improvement across all 48 PCA andKKsettings fortext\-embedding\-3\-largein both datasets\. These results evaluate each induced taxonomy as a whole, including its grouping of questions, number of categories, label frequencies, and model parameters tied to its labels\. They do not show which property drives the gain, and AUC measures prediction rather than educational value\. After accounting for PCA dimension andKK, closer agreement with the original taxonomy is not consistently associated with higher AUC\. The negative correlations in 2017 therefore show that lower agreement is sometimes associated with higher AUC, not that disagreement is inherently beneficial\.

The induced taxonomies are therefore best treated as complements to systems developed by experts\. Their similarities and differences can highlight categories or boundaries for expert review\. Determining whether these distinctions matter for teaching or reflect how students reason requires evidence from educators, student behavior, and solution steps\.

A comparison with biological taxonomy helps clarify the idea\. Biological categories are sometimes reconsidered using genetic evidence alongside visible traits[San Mauro and Agorreta \(2010\)](https://arxiv.org/html/2610.33694#bib.bib24)\. In the same way, text embeddings provide another source of evidence that may support some expert categories and question others\. The comparison has a clear limit: similar question text is not automatically a better basis for curriculum design\. Educational categories are useful when they support teaching, assessment, or models of learning\.

## 6Future Work

Future work should extend the analysis to datasets that combine student interactions, question text, and more detailed KC structures\. XES3G5M contains 5\.5 million responses, assigns multiple KCs to some questions, and links broader skills to narrower ones\. DBE\-KT22 provides question text and student responses from a university database course, assigns up to four KCs per question, and includes relationships among KCs[Liu et al\. \(2023\)](https://arxiv.org/html/2610.33694#bib.bib14);[Abdelrahman et al\. \(2022\)](https://arxiv.org/html/2610.33694#bib.bib1)\. These datasets would support induced taxonomies with multiple labels per question or hierarchical relations among categories\. The current ASSISTments datasets would remain useful for evaluating models that explicitly support these richer structures\. The induced taxonomies could also be evaluated in other tasks that use KCs, including measuring progress with practice, identifying students’ relative strengths and weaknesses, and recommending the next practice question[Wang et al\. \(2020\)](https://arxiv.org/html/2610.33694#bib.bib27);[Cen et al\. \(2006\)](https://arxiv.org/html/2610.33694#bib.bib6);[Ai et al\. \(2019\)](https://arxiv.org/html/2610.33694#bib.bib3)\.

## 7Conclusion

We use embeddings of question text to build educational taxonomies and compare them with the original KC taxonomies in ASSISTments 2012 and 2017\. The induced taxonomies preserve much, but not all, of the original structure, with closer agreement in 2012 and consistent changes as the number of induced categories increases\. In DKT, the induced taxonomy with the highest observed test AUC improves test AUC by \.0474 in 2012 and \.0272 in 2017, while closer agreement with the original taxonomy is not consistently associated with higher AUC across the tested settings\. The central result therefore separates two questions: similarity to a taxonomy developed by experts and usefulness for prediction\. Taxonomies built from embeddings offer a promising alternative for student models, but their educational value still requires review by experts and evidence from students\.

## 8Limitations

Our analysis covers only ASSISTments 2012 and 2017 questions with linked text\. Both datasets come from the same learning platform, so the findings may not extend to other platforms, courses, schools, languages, or educational settings\. We include only responses tied to questions with text\.

The procedure for building the taxonomy assigns each question to one induced category\. It does not allow several KCs per question, organize KCs from broad to narrow, or model student reasoning\. The induced taxonomy may organize questions by wording, templates, formatting, answer type, or source rather than by the mathematics or knowledge required to answer them\.

The comparison measures show how closely the two taxonomies group the same questions rather than their educational value\. Purity and homogeneity can rise simply with the number of categories, and overall scores do not identify which specific splits or mergers matter for instruction\. We do not analyze solution steps, inspect individual category boundaries, or ask educators to judge the categories, so we cannot determine the educational value of those differences\.

The experiments use two models for embedding text, one clustering method, one clustering seed, and a fixed set of PCA dimensions and values ofKK\. The observed results may therefore depend on the tested models, clustering procedure, category count, question templates, or distribution of cluster sizes\. We use PCA as the only dimensionality\-reduction method; alternative reduction methods may produce different induced taxonomies\. We do not directly benchmark against prior KC\-discovery methods such as SMART or KCluster\.

We evaluate one DKT architecture\. Although all taxonomies use the same records, student split, training procedure, search protocol, and trial budget, we tune DKT separately for each taxonomy, so the selected hyperparameters may differ\. ChangingKKalso changes the number of KC labels and can change the number of model parameters tied to those labels\. Without random\-label baselines matched on the number and sizes of categories, the experiment cannot separate grouping by meaning from the effects of category count, how often each label appears, or model size\.

The reported induced AUC is the highest result across all tested combinations of embedding model, PCA dimension, andKK\. Because this combination is identified using test AUC, the reported gain summarizes the best tested result rather than a combination chosen before looking at test performance\. Figure[4](https://arxiv.org/html/2610.33694#S4.F4)shows the results across the full grid\. For each combination, the Bayesian search chooses DKT settings using validation AUC and does not choose the embedding model, PCA dimension, orKK\. Because students rather than questions are held out, test AUC measures performance for new students on questions used to build the taxonomy, not entirely new questions\. LargerKKmay also make induced labels behave more like coarse question identifiers, which could contribute to higher AUC\. Finally, the analysis of taxonomy agreement and AUC shows an association and does not establish that changing agreement causes the AUC gains\.

## Acknowledgments

We thank the ASSISTments Project at Worcester Polytechnic Institute for providing the Problem Bodies data, which allowed us to link records of student responses with question text[ASSISTments Project \(n\.d\.\)](https://arxiv.org/html/2610.33694#bib.bib4)\.

## References

- Abdelrahman et al\. \(2022\)Ghodai Abdelrahman, Sherif Abdelfattah, Qing Wang, and Yu Lin\. 2022\.[DBE\-KT22: A knowledge tracing dataset based on online student evaluation](https://arxiv.org/abs/2208.12651)\.*arXiv preprint arXiv:2208\.12651*\.
- Abdelrahman et al\. \(2023\)Ghodai Abdelrahman, Qing Wang, and Bernardo Nunes\. 2023\.[Knowledge tracing: A survey](https://doi.org/10.1145/3569576)\.*ACM Computing Surveys*, 55\(11\):224:1–224:37\.
- Ai et al\. \(2019\)Fangzhe Ai, Yishuai Chen, Yuchun Guo, Yongxiang Zhao, Zhenzhu Wang, Guowei Fu, and Guangyan Wang\. 2019\.Concept\-aware deep knowledge tracing and exercise recommendation in an online learning system\.In*Proceedings of the 12th International Conference on Educational Data Mining*, pages 240–245\. International Educational Data Mining Society\.
- ASSISTments Project \(n\.d\.\)ASSISTments Project\.[ASSISTments Problems](https://sites.google.com/site/assistmentsdata/datasets/assistments-problems)\[online\]\. n\.d\.Accessed August 1, 2026\.
- Benjamini and Hochberg \(1995\)Yoav Benjamini and Yosef Hochberg\. 1995\.[Controlling the false discovery rate: A practical and powerful approach to multiple testing](https://doi.org/10.1111/j.2517-6161.1995.tb02031.x)\.*Journal of the Royal Statistical Society: Series B \(Methodological\)*, 57\(1\):289–300\.
- Cen et al\. \(2006\)Hao Cen, Kenneth Koedinger, and Brian Junker\. 2006\.[Learning factors analysis – a general method for cognitive model evaluation and improvement](https://doi.org/10.1007/11774303_17)\.In*Intelligent Tutoring Systems*, pages 164–175, Berlin, Heidelberg\. Springer\.
- Common Core State Standards Initiative \(n\.d\.\)Common Core State Standards Initiative\.[Development process](https://www.thecorestandards.org/about-the-standards/development-process/5/)\[online\]\. n\.d\.Accessed July 31, 2026\.
- Corbett and Anderson \(1995\)Albert T\. Corbett and John R\. Anderson\. 1995\.[Knowledge tracing: Modeling the acquisition of procedural knowledge](https://doi.org/10.1007/BF01099821)\.*User Modeling and User\-Adapted Interaction*, 4\(4\):253–278\.
- Feng et al\. \(2009\)Mingyu Feng, Neil Heffernan, and Kenneth Koedinger\. 2009\.[Addressing the assessment challenge with an online system that tutors as it assesses](https://doi.org/10.1007/s11257-009-9063-7)\.*User Modeling and User\-Adapted Interaction*, 19\(3\):243–266\.
- Fowlkes and Mallows \(1983\)Edward B\. Fowlkes and Colin L\. Mallows\. 1983\.[A method for comparing two hierarchical clusterings](https://doi.org/10.1080/01621459.1983.10478008)\.*Journal of the American Statistical Association*, 78\(383\):553–569\.
- Koedinger et al\. \(2012\)Kenneth R\. Koedinger, Albert T\. Corbett, and Charles Perfetti\. 2012\.[The knowledge–learning–instruction framework: Bridging the science–practice chasm to enhance robust student learning](https://doi.org/10.1111/j.1551-6709.2012.01245.x)\.*Cognitive Science*, 36:757–798\.
- Liu et al\. \(2021\)Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu\. 2021\.[EKT: Exercise\-aware knowledge tracing for student performance prediction](https://doi.org/10.1109/TKDE.2019.2924374)\.*IEEE Transactions on Knowledge and Data Engineering*, 33\(1\):100–115\.
- Liu et al\. \(2022\)Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Jiliang Tang, and Weiqi Luo\. 2022\.[pyKT: A Python library to benchmark deep learning based knowledge tracing models](https://doi.org/10.52202/068431-1347)\.In*Advances in Neural Information Processing Systems*, volume 35, pages 18542–18555\. Curran Associates, Inc\.
- Liu et al\. \(2023\)Zitao Liu, Qiongqiong Liu, Teng Guo, Jiahao Chen, Shuyan Huang, Xiangyu Zhao, Jiliang Tang, Weiqi Luo, and Jian Weng\. 2023\.[XES3G5M: A knowledge tracing benchmark dataset with auxiliary information](https://doi.org/10.52202/075280-1429)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 32958–32970\.
- Matsuda et al\. \(2022\)Noboru Matsuda, Jesse Wood, Raj Shrivastava, Machi Shimmei, and Norman Bier\. 2022\.[Latent skill mining and labeling from courseware content](https://doi.org/10.5281/zenodo.7086211)\.*Journal of Educational Data Mining*, 14\(2\):1–31\.
- Meilă \(2007\)Marina Meilă\. 2007\.[Comparing clusterings, an information based distance](https://doi.org/10.1016/j.jmva.2006.11.013)\.*Journal of Multivariate Analysis*, 98\(5\):873–895\.
- Moon et al\. \(2025\)Hyeongdon Moon, Richard Lee Davis, Seyed Parsa Neshaei, and Pierre Dillenbourg\. 2025\.[Using large multimodal models to extract knowledge components for knowledge tracing from multimedia question information](https://doi.org/10.5281/zenodo.15870171)\.In*Proceedings of the 18th International Conference on Educational Data Mining*, pages 342–353\. International Educational Data Mining Society\.
- OpenAI \(2026\)OpenAI\. 2026\.[Vector embeddings](https://developers.openai.com/api/docs/guides/embeddings)\.Accessed 2026\-08\-07\.
- Piech et al\. \(2015\)Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, and Jascha Sohl\-Dickstein\. 2015\.Deep knowledge tracing\.In*Advances in Neural Information Processing Systems*, volume 28, pages 505–513\.
- Porter et al\. \(2011\)Andrew Porter, Jennifer McMaken, JeongSuk Hwang, and Rui Yang\. 2011\.[Common core standards: The new U\.S\. intended curriculum](https://doi.org/10.3102/0013189X11405038)\.*Educational Researcher*, 40\(3\):103–116\.
- Prihar et al\. \(2022\)Ethan Prihar, Manaal Syed, Korinn Ostrow, Stacy Shaw, Adam Sales, and Neil Heffernan\. 2022\.[Exploring common trends in online educational experiments](https://doi.org/10.5281/zenodo.6853041)\.
- Reimers and Gurevych \(2019\)Nils Reimers and Iryna Gurevych\. 2019\.[Sentence\-BERT: Sentence embeddings using siamese BERT\-networks](https://doi.org/10.18653/v1/D19-1410)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing*, pages 3982–3992\.
- Rosenberg and Hirschberg \(2007\)Andrew Rosenberg and Julia Hirschberg\. 2007\.[V\-measure: A conditional entropy\-based external cluster evaluation measure](https://aclanthology.org/D07-1043/)\.In*Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning*, pages 410–420\. Association for Computational Linguistics\.
- San Mauro and Agorreta \(2010\)Diego San Mauro and Ana Agorreta\. 2010\.[Molecular systematics: A synthesis of the common methods and the state of knowledge](https://doi.org/10.2478/s11658-010-0010-8)\.*Cellular and Molecular Biology Letters*, 15\(2\):311–341\.
- Su et al\. \(2018\)Yu Su, Qingwen Liu, Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Chris Ding, Si Wei, and Guoping Hu\. 2018\.[Exercise\-enhanced sequential modeling for student performance prediction](https://doi.org/10.1609/aaai.v32i1.11864)\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 32\(1\):2435–2443\.
- Vinh et al\. \(2010\)Nguyen Xuan Vinh, Julien Epps, and James Bailey\. 2010\.[Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance](https://www.jmlr.org/papers/v11/vinh10a.html)\.*Journal of Machine Learning Research*, 11\(95\):2837–2854\.
- Wang et al\. \(2020\)Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang\. 2020\.[Neural cognitive diagnosis for intelligent education systems](https://doi.org/10.1609/aaai.v34i04.6080)\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pages 6153–6161\.
- Wei et al\. \(2025\)Yumou Wei, Paulo Carvalho, and John Stamper\. 2025\.[KCluster: An LLM\-based clustering approach to knowledge component discovery](https://doi.org/10.5281/zenodo.15870197)\.In*Proceedings of the 18th International Conference on Educational Data Mining*, pages 228–240\. International Educational Data Mining Society\.
- Xiong et al\. \(2016\)Xiaolu Xiong, Siyuan Zhao, Eric G\. Van Inwegen, and Joseph E\. Beck\. 2016\.Going deeper with deep knowledge tracing\.In*Proceedings of the 9th International Conference on Educational Data Mining*, pages 545–550\. International Educational Data Mining Society\.

## Appendix AQuestion Text Inputs and Examples

#### Question examples\.

The embedding models use only the cleaned text from theproblem\_bodyfield inquestions\.csv; grade information and the original category labels are excluded\. The grade labels range from elementary school through high school mathematics and do not necessarily match a student’s enrolled grade\. Although Grades 6–8 usually correspond to U\.S\. middle school, the data cover a broader range\. The Grade 4 and Grade 6 examples also illustrate that converting web pages to plain text can remove mathematical layout and diagrams\. Images are not provided to the models\. Because the access terms for Problem Bodies prohibit quoting the questions, we describe representative examples instead[ASSISTments Project \(n\.d\.\)](https://arxiv.org/html/2610.33694#bib.bib4)\.

- •ASSISTments 2012, Grade 4:converting an improper fraction\.
- •ASSISTments 2012, Grade 6:finding a square’s area from a diagram\.
- •ASSISTments 2017, Grade 7:finding a percentage of a whole\.
- •ASSISTments 2017, Grade 8:using the formula for a cone’s volume\.
- •ASSISTments 2012, High School Algebra:comparing the degrees of polynomial terms\.

## Appendix BTaxonomy and DKT Settings

By default, OpenAI’stext\-embedding\-3\-smallandtext\-embedding\-3\-largeproduce embeddings with 1,536 and 3,072 dimensions, respectively\. We use the API’sdimensionsparameter to request outputs with 768 and 1,536 dimensions, respectively[OpenAI \(2026\)](https://arxiv.org/html/2610.33694#bib.bib18)\. Leteie\_\{i\}denote the embedding for questionii\. PCA reduces it tozi=P⁡\(ei\)z\_\{i\}=P\(e\_\{i\}\), andkk\-means assignsziz\_\{i\}to the nearest centroid:

ci=arg⁡minj∈\{1,…,K\}⁡∥zi−μj∥22\.c\_\{i\}=\\arg\\min\_\{j\\in\\\{1,\\dots,K\\\}\}\\lVert z\_\{i\}\-\\mu\_\{j\}\\rVert\_\{2\}^\{2\}\.Here,cic\_\{i\}is the label assigned by the induced taxonomy,KKis the number of induced categories, andμj\\mu\_\{j\}is the centroid of clusterjj\. We test PCA dimensions\{10,20,40,60,80,100,120,160\}\\\{10,20,40,60,80,100,120,160\\\}\. We test values ofKK\{50,100,150,200,250,300\}\\\{50,100,150,200,250,300\\\}for 2012 and\{20,40,60,80,100,150\}\\\{20,40,60,80,100,150\\\}for 2017\. PCA andkk\-means use a random seed of 42\. For each setting,kk\-means uses 10 initializations and at most 300 iterations \(n\_init=10,max\_iter=300\)\.

For responsett,qtq\_\{t\}denotes the question,sts\_\{t\}its original KC label,rtr\_\{t\}whether the response is correct, andc⁡\(qt\)c\(q\_\{t\}\)the label assigned by the induced taxonomy\. We apply the relabeling

\(qt,st,rt\)↦\(qt,c⁡\(qt\),rt\)\.\(q\_\{t\},s\_\{t\},r\_\{t\}\)\\mapsto\(q\_\{t\},c\(q\_\{t\}\),r\_\{t\}\)\.For 2012, we write induced labels toskill\_id\. For 2017, we write them as text inskill\. pyKT then renumbers them as consecutive concept IDs\.

For the baseline that uses the original taxonomy and for every induced taxonomy, we run a separate Weights & Biases Bayesian search that maximizes validation AUC\. Each induced taxonomy is defined by its dataset, embedding model, PCA dimension, andKK\. Within each dataset, all searches use the same candidate settings, number of trials, assignment of students to training, validation, and testing, training procedure, and selection rule\.

For 2012, the search variesemb\_sizeover\{64,128,256,512\}\\\{64,128,256,512\\\},learning\_rateover\{0\.01,0\.001,0\.0007,0\.0005\}\\\{0\.01,0\.001,0\.0007,0\.0005\\\},dropoutover\{0\.1,0\.2,0\.3,0\.5\}\\\{0\.1,0\.2,0\.3,0\.5\\\}, andseedover\{42,84\}\\\{42,84\\\}\. For 2017, the search variesemb\_sizeover\{64,96,128,192,256,384,512,768,1024\}\\\{64,96,128,192,256,384,512,768,1024\\\},learning\_rateover\{0\.01,0\.001,0\.0007,0\.0005,0\.002,0\.0002\}\\\{0\.01,0\.001,0\.0007,0\.0005,0\.002,0\.0002\\\}, anddropoutover\{0\.01,0\.1,0\.2,0\.3,0\.4,0\.45,0\.5\}\\\{0\.01,0\.1,0\.2,0\.3,0\.4,0\.45,0\.5\\\}\. The 2017 search fixesseedat 42\. For 2012, seed is one of the settings considered by the search; it is not used to repeat each taxonomy\.

Within each search, we select the completed run with the highest validation AUC and report its test AUC; the embedding model, PCA dimension, andKKremain fixed outside the search\.

Both the DKT/QID input format, which includes question and KC identifiers, and fold 0 are fixed across all searches\. Training uses Adam with batch size 256, at most 200 epochs, and a maximum sequence length of 200\. The preprocessing code reserves 20% of students for testing withrandom\_state=1024; the remaining students are divided into five folds, yielding the approximate 64/16/20 training/validation/test split\. Training stops after 30 epochs without a validation AUC improvement of at least \.001\.

## Appendix CMeasures for Comparing the Taxonomies

Combining eight PCA dimensions with six values ofKKgives 48 settings per dataset\. We evaluate both embedding models at every setting, yielding 96 rows per dataset\. Each analysis below states whether results for the two embedding models are reported separately or averaged\.

Table 3:Measures used to compare the original and induced taxonomies[Rosenberg and Hirschberg \(2007\)](https://arxiv.org/html/2610.33694#bib.bib23);[Vinh et al\. \(2010\)](https://arxiv.org/html/2610.33694#bib.bib26);[Meilă \(2007\)](https://arxiv.org/html/2610.33694#bib.bib16);[Fowlkes and Mallows \(1983\)](https://arxiv.org/html/2610.33694#bib.bib10)\. IncreasingKKcan raise some scores and lower others, so the scores should be interpreted together\.Table 4:Best agreement scores across the 48 settings, averaged across the two embedding models\. ARI is the adjusted Rand index; AMI and NMI denote adjusted and normalized mutual information, respectively; FM is Fowlkes–Mallows; and VI is variation of information\. Parentheses give the PCA dimension andKK\. Lower VI means a closer match; higher values mean a closer match for the other measures\. NMI and V\-measure have the same values here and reach their highest values at the same setting\. Table[6](https://arxiv.org/html/2610.33694#A3.T6)shows these and the purity measures at the smallest and largest values ofKK\. Figure[6](https://arxiv.org/html/2610.33694#A3.F6)shows ARI and NMI separately for each embedding model and dataset\.Table 5:Structural scores at the two PCA/KKsettings used for the reported DKT results\. Each score is averaged across the two embedding models\. “Homog\.” means homogeneity, “Complete\.” means completeness, “pur\.” means purity, and “inv\.” means inverse purity\. Table[3](https://arxiv.org/html/2610.33694#A3.T3)defines ARI, AMI, and NMI; FM means Fowlkes–Mallows\. VI uses natural logarithms and is the only score for which lower values mean a closer match\. V\-measure and NMI have the same values here\.At the two reported combinations of PCA dimension andKK, the average spread in cluster size across the two embedding models is \.591 for 2012 and \.447 for 2017\. This spread is the standard deviation divided by the mean; a larger value means the cluster sizes vary more\. The average share of questions in the largest cluster is 1\.28% for 2012 and 2\.51% for 2017\. Neither 2012 embedding model produces a cluster with only one question; each 2017 embedding model produces one such cluster\. No model places more than 3% of questions in its largest cluster\. AllKKclusters contain at least one question\. The mean cluster size is 62\.45 for 2012 and 6\.38 for 2017\.

![Refer to caption](https://arxiv.org/html/2610.33694v1/assist2017_contingency_heatmap.png)Figure 5:Distribution of questions from the 25 largest categories in the 2017 original taxonomy across the corresponding induced taxonomy\. The figure usestext\-embedding\-3\-large, PCA=10=10, andK=100K=100\. The left heatmap shows question counts and the right heatmap shows the share of each original category in each induced category\. We include the three induced categories containing the most questions from each original category and combine the rest asOther\. Induced category identifiers are arbitrary labels and do not indicate an order\.\(a\) ASSISTments 2012

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/appendix/chapter6_cluster2012_metric_heatmaps.png)

\(b\) ASSISTments 2017

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/appendix/chapter6_cluster2017_metric_heatmaps.png)

Figure 6:ARI and NMI for every PCA dimension and value ofKK, shown separately for each embedding model in 2012 \(a\) and 2017 \(b\)\. Table[4](https://arxiv.org/html/2610.33694#A3.T4)gives the highest values after averaging scores across the two models\. In 2017, ARI and NMI reach their highest values at different settings, and, for each metric, each embedding model’s highest value differs from the highest value obtained after averaging scores across the two models\.\(a\) ASSISTments 2012

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/appendix/chapter6_cluster2012_homogeneity_completeness.png)

\(b\) ASSISTments 2017

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/appendix/chapter6_cluster2017_homogeneity_completeness.png)

Figure 7:Homogeneity and completeness across values ofKK, shown separately for each combination of embedding model and PCA dimension in 2012 \(a\) and 2017 \(b\)\. In 2012, homogeneity generally rises while completeness falls\. In 2017, both generally rise across the tested settings, although global inverse purity and pairwise recall fall from the smallest to the largest tested value ofKK, as shown in Table[6](https://arxiv.org/html/2610.33694#A3.T6)\. Outlined circles mark the smallest testedKK, and squares mark the largest\.Table 6:Scores at the smallest and largest tested values ofKK\. Each value is the simple average over eight PCA dimensions and both embedding models\. In both datasets, increasingKKraises homogeneity, global purity, and pairwise precision\. It lowers global inverse purity and pairwise recall\. In 2012, completeness falls and VI rises\. In 2017, completeness rises and VI falls\. “Homog\.” means homogeneity, “Complete\.” means completeness, and VI means variation of information\. Size spread is the standard deviation of the induced cluster sizes divided by their mean; a larger value means that the cluster sizes differ more\.
## Appendix DRelation Between Taxonomy Agreement and AUC

To examine the relationship between taxonomy agreement and AUC, we analyze 48 matched settings of PCA dimension andKKper dataset\. At each setting, we average each structural score across the two embedding models and separately average the two embedding\-specific test AUC values\. The 2017 AUC results include three additional settings,\(10,200\)\(10,200\),\(10,250\)\(10,250\), and\(10,300\)\(10,300\); we exclude them because the taxonomy comparison has no matching scores\.

We use Spearman correlation to test whether closer taxonomy agreement is associated with higher test AUC\. We then repeat the analysis after ranking both measures and removing patterns linked to PCA dimension andKK\. Because we test 15 measures, we apply the Benjamini–Hochberg correction separately by dataset and separately before and after this adjustment\.

![Refer to caption](https://arxiv.org/html/2610.33694v1/figures/appendix/metric_auc_correlation_heatmap.png)Figure 8:Rank correlations between each taxonomy\-agreement score and test AUC, both averaged across the two embedding models\. “After” indicates that rank patterns linked to PCA dimension andKKhave been removed\. Cells show coefficients; Table[7](https://arxiv.org/html/2610.33694#A4.T7)reports significance after correction\.Table 7:Associations, across the 48 shared settings of PCA dimension andKKin each dataset, between each taxonomy\-agreement score and test AUC, both averaged across the two embedding models\. “Before” is the direct Spearman correlation; “After” is the correlation after removing rank patterns linked to PCA dimension andKK\. An asterisk marksp<\.05p<\.05after the Benjamini–Hochberg correction, applied separately by dataset and analysis stage\. VI is a distance measure, so a positive coefficient indicates that a greater difference between the taxonomies is associated with higher test AUC\.