Beyond Top Words: MonoTM for Topic Modeling with Interpretable Monosemantic Features

# Beyond Top Words: MonoTM for Topic Modeling with Interpretable Monosemantic Features
Source: [https://arxiv.org/html/2609.09575](https://arxiv.org/html/2609.09575)
Bei YuAffiliation:School of Information StudiesAffiliation:Syracuse UniversityEmail:[\{sjoh01,byu\}@syr\.edu](mailto:)

###### Abstract

Topic models summarize large text corpora, but top\-ranked words often provide only a limited representation of topic semantics\. Sparse autoencoders \(SAEs\) offer a way to move beyond word\-level descriptors by extracting interpretable features from dense representations, yet how feature interpretability relates to topic\-inference quality remains unclear\. We introduceMonoTM, an interpretable topic modeling framework that decouples these roles\. Across three benchmark corpora, we show that document–topic mixture estimation and semantic interpretation favor different SAE configurations and feature subsets\. MonoTM estimates mixtures from the full SAE bag\-of\-features representation and, with them fixed, learns topic descriptors over a separate vocabulary of corpus\-grounded semantic features\. This design preserves global topic structure while representing topics with semantic units more meaningful than individual words, making them more useful for downstream corpus analysis\.

## 1Introduction

Text is a primary source of evidence across the social sciences and the humanities, from political speeches and news coverage to historical archives and literary corpora\. The scale of these collections motivates computational text\-as\-data methods; topic modeling, especially Latent Dirichlet Allocation \(LDA\) and its extensions, has therefore become a popular tool for exploratory corpus mapping and thematic summarization\([Blei et al\., 2003](https://arxiv.org/html/2609.09575#bib.bib4)\)\. Yet its role as scholarly evidence remains contested[Da \(2019\)](https://arxiv.org/html/2609.09575#bib.bib7);[Grimmer and Stewart \(2013\)](https://arxiv.org/html/2609.09575#bib.bib5);[Shadrova \(2021\)](https://arxiv.org/html/2609.09575#bib.bib6)\. A recent review of topic modeling validation in computational social science finds that validation and reporting practices remain heterogeneous and show little convergence across studies\([Bernhard\-Harrer et al\., 2025](https://arxiv.org/html/2609.09575#bib.bib8)\)\.

One source of this issue is that most topic models define each topic as a probability distribution over words, and users typically interpret topics by their highest\-probability words\. However, assigning a concise, human\-readable meaning or label to each inferred topic can be challenging in practice\([Chang et al\., 2009](https://arxiv.org/html/2609.09575#bib.bib9);[Mei et al\., 2007](https://arxiv.org/html/2609.09575#bib.bib10)\)\. Although recent neural topic models incorporate pretrained language models or document embeddings to capture semantic information beyond raw word counts, these models still typically extract top\-ranked words or phrases post hoc to describe each topic to users\([Bianchi et al\., 2021b](https://arxiv.org/html/2609.09575#bib.bib11);[Dieng et al\., 2020](https://arxiv.org/html/2609.09575#bib.bib12);[Grootendorst, 2022](https://arxiv.org/html/2609.09575#bib.bib2);[Kardos et al\., 2025](https://arxiv.org/html/2609.09575#bib.bib13)\)\.

This creates a gap between what topic models output—topics as distributions over words—and what many analysts want: semantic units that support interpretation at an intermediate level between reading individual documents and identifying coarse corpus\-level themes\. Beyond topic representation, words are rarely adequate semantic units for supporting meaningful downstream corpus analysis, such as analyzing relations between topics or documents based on word overlap\.

Recent progress in mechanistic interpretability, particularly sparse autoencoders \(SAEs\) and autointerpretability methods, offers a way to extract human\-readable semantic features from document embeddings produced by pretrained encoders\. Motivated by our experimental findings, we proposeMonoTM, an interpretable topic modeling framework that achieves competitive document\-\-topic mixture estimation while producing validated, human\-readable topic descriptors\.111The official MonoTM repository is available at https://github\.com/Una\-J/MonoTM

## 2Background & Related Work

### 2\.1Mechanistic Interpretability and Sparse Autoencoders

Mechanistic interpretability seeks to explain neural networks by identifying internal components and how they compose into computations\. A major obstacle is that neurons are often polysemantic, activating for multiple unrelated patterns[Olah et al\. \(2020\)](https://arxiv.org/html/2609.09575#bib.bib17)\. One influential account explains this via superposition, where models represent more features than they have dimensions by encoding features in an overcomplete set of directions, enabled by sparsity in the underlying factors\([Elhage et al\., 2022](https://arxiv.org/html/2609.09575#bib.bib16)\)\. This motivates moving from neurons to learned feature directions\.

Two scalable strategies support this shift\. First,[Bills et al\. \(2023\)](https://arxiv.org/html/2609.09575#bib.bib18)employ language models to produce and score natural\-language explanations of internal directions based on activation patterns\. Second, sparse dictionary learning methods, especially sparse autoencoders \(SAEs\), learn sparse, overcomplete feature bases that often yield more monosemantic units than neurons and can support meaningful interventions\([Bricken et al\., 2023](https://arxiv.org/html/2609.09575#bib.bib14);[Huben et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib15)\)\. Recent work also applies SAEs to dense text embeddings, recovering interpretable semantic factors useful for downstream control and analysis\([O’Neill et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib1)\)\.

Building on these ideas, we investigate SAEs over document embeddings as a source of reusable semantic units for topic modeling, and adopt an Interpreter–Predictor style protocol to obtain validated feature labels for topic description\.

### 2\.2Topic Modeling

##### Overview of topic models\.

Topic modeling began with probabilistic bag\-of\-words \(BoW\) generative models such as LDA\([Blei et al\., 2003](https://arxiv.org/html/2609.09575#bib.bib4)\), which represents each document as a mixture of topics and each topic as a distribution over words\. Although LDA offers clear probabilistic semantics and interpretable topic descriptors, its BoW assumption ignores context and can produce frequency\- and surface\-form\-sensitive topics\. Neural topic models use inference networks for amortized variational inference instead of iterative per\-document inference \(ProdLDA;\([Srivastava and Sutton, 2017](https://arxiv.org/html/2609.09575#bib.bib19)\)\)\. Other neural parameterizations of document–topic mixtures include softmax and stick\-breaking constructions\([Miao et al\., 2017](https://arxiv.org/html/2609.09575#bib.bib20)\)\. In parallel, embedding\-aware models such as ETM represent topics in a word\-embedding space to improve robustness to large vocabularies while maintaining interpretability\([Dieng et al\., 2020](https://arxiv.org/html/2609.09575#bib.bib12)\)\.

More recent approaches incorporate pretrained transformers to inject contextual semantics\. A representative direction is contextualized topic modeling\. CombinedTM uses contextual embeddings alongside BoW\-style signals to improve topic quality\([Bianchi et al\., 2021b](https://arxiv.org/html/2609.09575#bib.bib11)\), and ZeroShotTM extends this idea with multilingual encoders to enable cross\-lingual, zero\-shot topic inference\([Bianchi et al\., 2021a](https://arxiv.org/html/2609.09575#bib.bib21)\)\. Beyond contextualization, recent work focuses on robustness and efficiency\. ECRTM introduces embedding clustering regularization to discourage topic collapse and encourage more distinct topics\([Wu et al\., 2023](https://arxiv.org/html/2609.09575#bib.bib22)\)\. FASTopic further leverages pretrained transformers while modeling document–topic and topic–word relations in a unified framework designed to be fast, stable, and transferable\([Wu et al\., 2024b](https://arxiv.org/html/2609.09575#bib.bib23)\)\. Finally, S3 offers a decomposition\-based alternative that treats topics as independent semantic axes in embedding space, discovered via ICA rather than clustering or neural decoders\([Kardos et al\., 2025](https://arxiv.org/html/2609.09575#bib.bib13)\)\.

##### SAE\-based topic modeling\.

Recent work connects sparse autoencoders and topic modeling by leveraging mechanistic\-interpretability insights and treating SAE latents as reusable semantic units in embedding/activation space\.[Girrbach and Akata \(2025\)](https://arxiv.org/html/2609.09575#bib.bib24)view SAEs as continuous\-space topic models by deriving the SAE objective as a MAP estimator under an LDA\-like generative model\.[Zheng et al\. \(2025\)](https://arxiv.org/html/2609.09575#bib.bib25)propose Mechanistic Topic Models \(MTMs\), which featurize documents using a pretrained SAE over LLM activations and fit topic models over SAE feature counts\. While their mLDA is closest to our LDA\-with\-bag\-of\-features approach, MTMs rely on pretrained SAEs and external feature descriptions\.

Unlike these prior works, our goal is not simply to replace words with SAE latents as the vocabulary for topic modeling\. We show that the features most useful for document–topic mixture estimation are not necessarily the same features whose labels can be reliably interpreted\. This empirical mismatch motivates MonoTM, which decouples the statistical and interpretive roles of SAE features\. Although this design is more costly than reusing pretrained SAE vocabularies, it targets settings such as social science and digital humanities, where corpus\-specific interpretability, traceability, and validation are often more important than fast topic discovery\.

## 3Research Questions

Our starting hypothesis is that SAE features can play two distinct roles in topic modeling\. If a document embedding can be represented as a sparse, nonnegative linear combination of SAE features, and if those features are closer to monosemantic than the original embedding dimensions, then SAE features may serve both as human\-readable semantic units and as useful units for estimating document–topic mixtures\. These two roles are related but not identical: features that are easy to label may not be the same features that best support mixture estimation\. We therefore first study these roles separately, and then ask how they can be combined in a single interpretable topic model:

1. 1\.Under what SAE configurations do corpus\-trained features become reliably interpretable semantic units?
2. 2\.Under what SAE configurations do SAE features provide effective units for document–topic mixture estimation?
3. 3\.How can we use SAE features to build a topic model that produces interpretable semantic topic descriptors aligned with strong document–topic mixture estimates?

## 4Experimental Setup

##### Datasets\.

To evaluate topic models with minimal subjectivity, we compare inferred topics against the gold category annotations provided by standard benchmarks \(dataset statistics in Table[3](https://arxiv.org/html/2609.09575#A1.T3)\)\. We use\(1\) 20 Newsgroups\(user\-generated Usenet newsgroup posts; scikit\-learn version\)\([Lang, 1995](https://arxiv.org/html/2609.09575#bib.bib27)\),\(2\) Web of Science\(WOS\-46985\) using abstracts as input\([Kowsari et al\., 2017](https://arxiv.org/html/2609.09575#bib.bib26)\), and\(3\) Reuters\(Reuters\-21578\) with the ModApte split\([Lewis, 1997](https://arxiv.org/html/2609.09575#bib.bib29)\)accessed via Hugging Face Datasets\([Lhoest et al\., 2021](https://arxiv.org/html/2609.09575#bib.bib28)\)\. Reuters is a multi\-label and highly imbalanced collection of Reuters newswire stories\. For Reuters, we keep labels appearing at least 30 times, yielding 47 topics\.

##### Document embeddings and SAE training\.

We map each documentddto a dense embeddingxd∈ℝ4096x\_\{d\}\\in\\mathbb\{R\}^\{4096\}using NVIDIA’sllama\-embed\-nemotron\-8bmodel[Babakhin et al\. \(2025\)](https://arxiv.org/html/2609.09575#bib.bib30)\. We use the model’s full context window of 32,768 tokens[NVIDIA \(2025\)](https://arxiv.org/html/2609.09575#bib.bib31); longer documents are truncated to their first 32,768 tokens\. This affects only 18 documents in 20 Newsgroups\.

For each corpus, we train top\-KKsparse autoencoders on dimension\-wise standardized document embeddings\. Each SAE learns a dictionary withm​NmNlatent features, wherem=4096m=4096is the embedding dimension andNNis the expansion factor\. Thus,NNcontrols the size of the learned feature dictionary, whileKKcontrols the maximum number of active features per document\. Given a document embedding, the encoder produces nonnegative latent activations and retains only the largestKK, yielding a sparse document–feature representation\. The decoder reconstructs the standardized embedding from these active features\.

The two SAE hyperparameters therefore have distinct roles:NNcontrols dictionary capacity, whileKKcontrols per\-document activity\. We sweep 11 values ofNNfrom1/641/64to55and eight values ofKKfrom44to512512, omitting configurations withm​N<KmN<K\. Full architectural, optimization, checkpoint\-selection, and grid details are given in Appendix[B](https://arxiv.org/html/2609.09575#A2)\.

##### Interpreter: feature label generation\.

Following and adapting the autointerpretability procedure of[O’Neill et al\. \(2024\)](https://arxiv.org/html/2609.09575#bib.bib1), we pass the learned SAE features \(decoder columns\) to an autointerpretability module that proposes and later validates natural\-language labels\.

For each featuref∈\{1,…,m​N\}f\\in\\\{1,\\dots,mN\\\}, we construct an Interpreter prompt using three sets of examples:Max\-activating examples, the 10 documents with the highest activationshd,fh\_\{d,f\};Typical\-activating examples, 10 documents sampled from the middle quantiles of the nonzero activation distribution forff; andZero\-activating examples, 10 documents where featureffis inactive \(hd,f=0h\_\{d,f\}=0\), sampled as negatives\.

Given these examples, an*Interpreter LLM*outputs a short descriptionℓf\\ell\_\{f\}\(e\.g\., 4–10 words\) intended to capture the single most prominent concept that is present in the activating texts but absent from the non\-activating texts \(see the Interpreter prompt in Figure[2](https://arxiv.org/html/2609.09575#A3.F2)\)\.

##### Predictor: feature label validation\.

Labels produced by an Interpreter LLM can be plausible but unvalidated\. For example, labels may describe a frequent corpus theme that is not specific to the feature\. To test whether a label actually predicts feature behavior, we validate eachℓf\\ell\_\{f\}using a separate Predictor LLM\([Bills et al\., 2023](https://arxiv.org/html/2609.09575#bib.bib18);[O’Neill et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib1)\)\.

For each featureff, we construct a balanced evaluation set𝒮f=𝒮f\+∪𝒮f−\\mathcal\{S\}\_\{f\}=\\mathcal\{S\}\_\{f\}^\{\+\}\\cup\\mathcal\{S\}\_\{f\}^\{\-\}by sampling 30*positive*documents and 30*negative*documents from the corpus\. Positives satisfyhd,f\>0h\_\{d,f\}\>0\(featureffis active for documentdd\), while negatives satisfyhd,f=0h\_\{d,f\}=0\(featureffis inactive\)\. Letyd,f=𝟏\[hd,f\>0\]y\_\{d,f\}=\\mathbf\{1\}\[h\_\{d,f\}\>0\]denote the ground\-truth activation indicator\.

We run the Predictor as an independent per\-document binary classification query\. For eachd∈𝒮fd\\in\\mathcal\{S\}\_\{f\}, the Predictor LLM receives only the descriptionℓf\\ell\_\{f\}and the raw document text, and outputs a binary predictiony^d,f∈\{0,1\}\\hat\{y\}\_\{d,f\}\\in\\\{0,1\\\}indicating whether featureffwould activate \(see the Predictor prompt in Figure[2](https://arxiv.org/html/2609.09575#A3.F2)\)\. We aggregate these document\-level predictions into a feature\-level interpretability score using the F1 score between\{y^d,f\}d∈𝒮f\\\{\\hat\{y\}\_\{d,f\}\\\}\_\{d\\in\\mathcal\{S\}\_\{f\}\}and\{yd,f\}d∈𝒮f\\\{y\_\{d,f\}\\\}\_\{d\\in\\mathcal\{S\}\_\{f\}\}, and denote this score byIS⁡\(f\)\\mathrm\{IS\}\(f\)\.

##### Autointerpretability sweep and LLM setup\.

Running the Interpreter–Predictor protocol requires many LLM inference calls, so we apply it only to a subset of SAE configurations we expect to be most informative due to cost constraints:N∈\{0\.5,1,2,3,4,5\}N\\in\\\{0\.5,1,2,3,4,5\\\}andK∈\{4,8,16,32\}K\\in\\\{4,8,16,32\\\}\. Details of the LLMs used are in Appendix[D](https://arxiv.org/html/2609.09575#A4)\.

## 5Results

### 5\.1RQ1: When Do SAE Features Become Interpretable Semantic Units?

RQ1 asks whether corpus\-trained SAE features can serve as reliable semantic units, and how this depends on SAE capacity and sparsity\. We operationalize interpretability using the Interpreter–Predictor protocol described in Section[4](https://arxiv.org/html/2609.09575#S4): a feature is treated as interpretable when its natural\-language label predicts held\-out feature activations with an interpretability score above a validation threshold\.

##### Empirical results\.

SAE features become interpretable semantic units most consistently under low\-to\-moderate per\-document activity levels\. Across all three datasets, configurations with smallerKKhave the highest fraction of activated features that pass validation \(Tables[4](https://arxiv.org/html/2609.09575#A5.T4),[5](https://arxiv.org/html/2609.09575#A5.T5), and[6](https://arxiv.org/html/2609.09575#A5.T6)\)\.

IncreasingKKchanges this behavior\. At permissive validation thresholds, largerKKgenerally increases the absolute number of validated features, because more active features are exposed for labeling and validation\. However, when averaged across theNNvalues we evaluate, the fraction of activated features that are validated decreases monotonically withKKon all datasets \(Tables[4](https://arxiv.org/html/2609.09575#A5.T4),[5](https://arxiv.org/html/2609.09575#A5.T5), and[6](https://arxiv.org/html/2609.09575#A5.T6)\)\.

The effect of dictionary capacityNNis best understood conditional on the activity levelKK\. On 20 Newsgroups and Reuters, increasingNNat lowKKoften reduces the number of activated features that surpass the activation threshold, while Web of Science is an exception: activated and validated counts increase withNNeven at lowKK\(Tables[4](https://arxiv.org/html/2609.09575#A5.T4),[5](https://arxiv.org/html/2609.09575#A5.T5), and[6](https://arxiv.org/html/2609.09575#A5.T6)\)\. This suggests that the value of increasing dictionary capacity depends on whether the corpus and activity level provide enough support for the larger feature space\. At higherKK, larger dictionaries are more often useful, as allowing documents to activate more features enables higher\-capacity SAEs to expose a larger pool of validated semantic units\.

Additionally, we report a secondary diagnostic analysis of feature granularity in Appendix[F](https://arxiv.org/html/2609.09575#A6)\. This analysis suggests that increasing capacity and activity may shift the validated feature space toward finer semantic resolution\.

##### Choosing\(N,K\)\(N,K\)in practice\.

These results suggest that SAE configuration should be treated as a practical modeling choice\. If the user wants a conservative, easy\-to\-audit feature vocabulary, the best starting point is a low\- or moderate\-activity SAE, such asK=4K=4orK=8K=8, combined with a moderately large dictionary\. This setting yields fewer features, but a larger share of them can be validated\. If the user instead wants broader coverage or more fine\-grained descriptors, it is reasonable to move toward largerKKand largerNN, but only with the expectation that post\-hoc validation will discard a larger fraction of the activated feature space\.

A useful practical procedure is therefore to begin with a high\-precision anchor configuration and then expand only if the validated feature inventory is too coarse or too small\. For corpora with many stable domain\-specific distinctions, such as Web of Science, a moderate configuration such asK=8K=8with a larger dictionary is a natural starting point\. For noisier or more heterogeneous corpora such as 20 Newsgroups and Reuters, low\-KKconfigurations provide a cleaner initial semantic inventory\.

### 5\.2RQ2: When Do SAE Features Support Document–Topic Mixture Estimation?

Parallel to RQ1’s focus on interpretability, RQ2 asks whether corpus\-trained SAE features can serve as useful statistical units for estimating document–topic mixtures\.

We test this hypothesis by replacing LDA’s word vocabulary with SAE features\. For each document, we treat active SAE features as pseudo\-tokens and their activation magnitudes as token weights, producing a bag\-of\-features \(BoF\) representation\. We deliberately choose LDA because it provides a transparent mixed\-membership likelihood once document embeddings are converted into bags of SAE features\. Neural topic models could also be applied to the same BoF inputs, but their neural parameterization would introduce additional representational capacity, making it harder to isolate whether the SAE feature vocabulary itself provides useful statistical units for topic inference\.

RQ2 evaluates whether SAE features are useful for mixture estimation, regardless of their interpretability scores\.

##### BoF\+LDA setup\.

For each dataset, we train a top\-KKSAE and construct a sparse document–feature matrixX∈ℝ≥0D×m​NX\\in\\mathbb\{R\}\_\{\\geq 0\}^\{D\\times mN\}whereXd,jX\_\{d,j\}is the activation magnitude of featurejjin documentdd\. We then fit LDA using scikit\-learn’sLatentDirichletAllocationwith variational EM for 200 iterations, and set the number of topics to the number of gold categories \(20 Newsgroups:T=20T\{=\}20, Web of Science:T=7T\{=\}7, Reuters:T=47T\{=\}47\)\.

##### Baselines\.

We compare against \(i\) classical BoW\-LDA\([Blei et al\., 2003](https://arxiv.org/html/2609.09575#bib.bib4)\)and \(ii\) other state\-of\-the\-art neural topic models that output document–topic mixtures: CombinedTM\([Bianchi et al\., 2021b](https://arxiv.org/html/2609.09575#bib.bib11)\), ZeroShotTM\([Bianchi et al\., 2021a](https://arxiv.org/html/2609.09575#bib.bib21)\), ECRTM\([Wu et al\., 2023](https://arxiv.org/html/2609.09575#bib.bib22)\), FASTopic\([Wu et al\., 2024b](https://arxiv.org/html/2609.09575#bib.bib23)\), and S3\([Kardos et al\., 2025](https://arxiv.org/html/2609.09575#bib.bib13)\)\. We do not include BERTopic because, following the taxonomy of[Wu et al\. \(2024a\)](https://arxiv.org/html/2609.09575#bib.bib3), it is a clustering\-based topic discovery method rather than a model for document–topic mixture estimation\. Including it in the topic–label alignment benchmark would therefore conflate mixture\-estimation quality with embedding\-cluster separability\.

For each baseline, we apply the same topic–label alignment procedure as for BoW\+LDA\. To ensure a fair comparison, all baselines that use document embeddings—CombinedTM, ZeroShotTM, FASTopic, and S3—are run with the samellama\-embed\-nemotron\-8bdocument embeddings used by MonoTM\.

##### Evaluation\.

We do not use standard word\-based topic coherence or topic diversity as evaluation metrics because doing so would defeat the purpose of MonoTM\. Metrics such as PMI, NPMI,CVC\_\{V\}, and topic diversity were designed for topic representations expressed as ranked lists of keywords: they measure whether top words co\-occur in a reference or training corpus, or whether top\-word lists are lexically redundant across topics\([Newman et al\., 2009](https://arxiv.org/html/2609.09575#bib.bib33);[Mimno et al\., 2011](https://arxiv.org/html/2609.09575#bib.bib34);[Lau et al\., 2014](https://arxiv.org/html/2609.09575#bib.bib35);[Röder et al\., 2015](https://arxiv.org/html/2609.09575#bib.bib36);[Dieng et al\., 2020](https://arxiv.org/html/2609.09575#bib.bib12)\)\. These metrics would require projecting our descriptors back into keywords that represent each topic, thereby reintroducing the very representation that our method is designed to replace\.

We instead evaluate document–topic mixture quality by aligning inferred topics with gold benchmark labels and reporting Micro\-F1 and Macro\-F1\. Full evaluation details, including the single\-label and multi\-label alignment procedures, are given in Appendix[G](https://arxiv.org/html/2609.09575#A7)\. For all models and datasets, we run three trials with different random seeds and report mean scores in Table[1](https://arxiv.org/html/2609.09575#S5.T1); full SAE\-grid heatmaps are reported in Appendix[H](https://arxiv.org/html/2609.09575#A8)\.

#### 5\.2\.1Effective mixture estimation requires balanced SAE scaling

BoF\+LDA performance depends systematically on the SAE hyperparameters\. The full Micro\-F1 and Macro\-F1 heatmaps are reported in Appendix[H](https://arxiv.org/html/2609.09575#A8)\. Across datasets, mixture quality is weakest in two regimes: very wide dictionaries paired with very smallKK, and boundary cases wherem​N=KmN=K\. These results suggest that SAE features are most useful for topic inference when the representation maintains an explicit sparsity bottleneck while still allowing enough active features per document\.

The best\-performing configurations instead lie in a balanced scaling band where dictionary capacity and per\-document activity increase together\. We therefore summarize BoF\+LDA in Table[1](https://arxiv.org/html/2609.09575#S5.T1)by averaging over a fixed robust subset of this region, usingN∈\{0\.25,0\.5,1,2\}N\\in\\\{0\.25,0\.5,1,2\\\}andK∈\{128,256,512\}K\\in\\\{128,256,512\\\}\. This gives a less brittle comparison than selecting the single best hyperparameter setting for each dataset\.

Model20 NewsgroupsWeb of ScienceReutersMicro F1Macro F1Micro F1Macro F1Micro F1Macro F1BoW\-LDA0\.36730\.35150\.51470\.47020\.34460\.2021CombinedTM0\.49290\.45130\.55140\.52010\.23980\.1728ZeroShotTM0\.37040\.33980\.49040\.46700\.20510\.1467ECRTM0\.57740\.53680\.51760\.49160\.24100\.1557FASTopic0\.54920\.49120\.50320\.45830\.25350\.1582S30\.45780\.42200\.42330\.39420\.15130\.1100BoF\+LDA \(Ours\)0\.67700\.64890\.57530\.54460\.31880\.2089Table 1:Micro F1 and Macro F1 scores on 20 Newsgroups, Web of Science, and Reuters datasets\. The best scores are in bold\.
#### 5\.2\.2Mixture quality and interpretability do not fully coincide

The balanced\-scaling pattern identified above should not be read as an interpretability result\. Although Figure[4](https://arxiv.org/html/2609.09575#A8.F4)identifies a robust region for document–topic mixture estimation, the same region is not necessarily the one that maximizes the number of interpretable SAE features\. This can be seen by comparing the mixture\-quality heatmaps in Figure[4](https://arxiv.org/html/2609.09575#A8.F4)with the interpretability counts in Tables[4](https://arxiv.org/html/2609.09575#A5.T4),[5](https://arxiv.org/html/2609.09575#A5.T5), and[6](https://arxiv.org/html/2609.09575#A5.T6)\. For example, in 20 Newsgroups, holdingN=0\.5N=0\.5fixed and increasingKKfrom88to3232reduces the fraction of dictionary features withIS≥0\.80\\mathrm\{IS\}\\geq 0\.80from721/\(4096⋅0\.5\)=35\.2%721/\(4096\\cdot 0\.5\)=35\.2\\%to282/\(4096⋅0\.5\)=13\.8%282/\(4096\\cdot 0\.5\)=13\.8\\%, while Micro/Macro F1 increases from 0\.56/0\.52 to 0\.66/0\.63\. Thus, neither the absolute number nor the dictionary\-level fraction of validated features fully predicts document–topic mixture quality\. These results suggest a tension between choosing SAE configurations for feature\-level interpretability and choosing them for document–topic mixture quality\.

We test this tension more directly by asking whether restricting BoF\+LDA to highly interpretable SAE features improves document–topic mixture estimation\. Specifically, we filter features according to their interpretability score, retain only those withIS⁡\(f\)≥τ\\mathrm\{IS\}\(f\)\\geq\\tau, and then re\-fit LDA on the resulting filtered BoF representation\. Figures[6](https://arxiv.org/html/2609.09575#A9.F6)–[8](https://arxiv.org/html/2609.09575#A9.F8)show that this filtering generally hurts document–topic mixture quality, and that the degradation becomes more severe asτ\\tauincreases\. These results indicate that failing strict interpretability validation does not imply that an SAE feature is uninformative for mixture estimation\.

#### 5\.2\.3Bag\-of\-features yields strong document–topic mixture estimates

Table[1](https://arxiv.org/html/2609.09575#S5.T1)shows that BoF\+LDA is competitive across all three benchmarks, achieving the strongest results on both single\-label datasets and the best Macro\-F1 on Reuters\. These gains support the core intuition that SAE latents provide a sparse, semantically structured pseudo\-token vocabulary, allowing LDA to pool co\-occurrence evidence more effectively than when using raw words\.

Results on Reuters are nuanced\. Since it is multi\-label and highly imbalanced, Micro\-F1 is dominated by frequent classes while Macro\-F1 is sensitive to performance on rare labels\. Here, BoW\-LDA attains the highest Micro\-F1, but BoF\+LDA achieves the best Macro\-F1 overall\. This pattern is consistent with BoF features providing more discriminative evidence for minority topics that have limited word overlap with majority classes\.

### 5\.3RQ3: Interpretable Topic Modeling \(MonoTM\)

RQ3 asks how the two roles of SAE features identified above can be combined into a single interpretable topic model\. The findings from RQ1 and RQ2 suggest that a single SAE representation should not be forced to serve both feature\-level interpretability and document–topic mixture estimation\. MonoTM implements this idea by separating document–topic mixture estimation from topic interpretation\.

The remainder of this section presents MonoTM in three steps\. Section[5\.3\.1](https://arxiv.org/html/2609.09575#S5.SS3.SSS1)describes the MonoTM algorithm\. Section[5\.3\.2](https://arxiv.org/html/2609.09575#S5.SS3.SSS2)audits whether the validated descriptor vocabulary omits important topic semantics from lower\-validation features\. Section[5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3)gives a compact descriptor comparison against word\-based and free\-form LLM descriptor baselines\.

#### 5\.3\.1The MonoTM Algorithm

MonoTM separates the statistical and interpretive roles of SAE features\. LetDDbe the number of documents,TTthe number of topics, andVVthe number of validated interpretable features used for topic description\. MonoTM estimates document–topic mixtures from a full SAE bag\-of\-features representation, then estimates a topic–feature distribution over validated interpretable features with the mixtures held fixed\.

##### Stage 1: estimating document–topic mixtures from all SAE features\.

For each document, the mixture SAE produces a sparse nonnegative activation vector over all SAE features\. We construct a full document–feature matrixXmixX^\{\\mathrm\{mix\}\}from these activations and fit LDA withTTtopics, yielding document–topic mixtures

Θ∈ℝ≥0D×T,θd∈ΔT\.\\Theta\\in\\mathbb\{R\}^\{D\\times T\}\_\{\\geq 0\},\\qquad\\theta\_\{d\}\\in\\Delta^\{T\}\.This stage uses all active SAE features, rather than only validated interpretable features, because Section[5\.2\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS2)shows that filtering to interpretable features degrades mixture quality\.

##### Stage 2: constructing an interpretable document–feature matrix\.

For topic interpretation, we use a validated feature setℱτ\\mathcal\{F\}\_\{\\tau\}\. A feature is retained if it activates in at least 30 documents, has a nonempty label, and satisfiesI​S​\(f\)≥τIS\(f\)\\geq\\tau, withτ=0\.8\\tau=0\.8in our experiments\. For each retained featureff, we construct an interpretable document–feature matrixC∈ℝ≥0D×VC\\in\\mathbb\{R\}^\{D\\times V\}\_\{\\geq 0\}by weighting its activation by its interpretability score:

cd,f=max⁡\(h~d,f,0\)⋅I​S​\(f\),f∈ℱτ\.c\_\{d,f\}=\\max\(\\tilde\{h\}\_\{d,f\},0\)\\cdot IS\(f\),\\qquad f\\in\\mathcal\{F\}\_\{\\tau\}\.

##### Stage 3: estimating topic–feature distributions with fixed mixtures\.

Given fixed document–topic mixturesΘ\\Thetaand interpretable feature matrixCC, MonoTM estimates a topic–feature distribution

B∈ℝ≥0T×V,βt∈ΔV,B\\in\\mathbb\{R\}^\{T\\times V\}\_\{\\geq 0\},\\qquad\\beta\_\{t\}\\in\\Delta^\{V\},by maximizing the fixed\-mixture likelihood

ℒ⁡\(B\)=∑d=1D∑f=1Vcd,f​log⁡\(∑t=1Tθd,t​βt,f\)\.\\mathcal\{L\}\(B\)=\\sum\_\{d=1\}^\{D\}\\sum\_\{f=1\}^\{V\}c\_\{d,f\}\\log\\left\(\\sum\_\{t=1\}^\{T\}\\theta\_\{d,t\}\\beta\_\{t,f\}\\right\)\.We optimize this objective with a Dirichlet\-smoothed EM procedure; full update equations and implementation details are given in Appendix[K](https://arxiv.org/html/2609.09575#A11)\.

After estimatingBB, each topicttis represented by the highest\-probability validated feature labels underβt\\beta\_\{t\}\. Appendix[J](https://arxiv.org/html/2609.09575#A10)provides sample MonoTM topic–feature representations produced by this final descriptor layer\.

#### 5\.3\.2Audit: do excluded features hide important topic semantics?

To test whether the validated descriptor vocabulary omits important topic semantics, we audit the excluded features most likely to matter for topic interpretation\. For all three datasets, we estimateΘ\\Thetafrom the full bag\-of\-features representation of an SAE withN=0\.5N=0\.5andK=256K=256\. For interpretation, we use theN=2,K=16N=2,K=16SAE\. The final MonoTM descriptor vocabulary contains active, validated features withIS⁡\(f\)≥\.8\\mathrm\{IS\}\(f\)\\geq\.8\. For the audit, we estimate an auxiliary all\-feature topic–feature distributionBallB^\{\\mathrm\{all\}\}over all active interpretation\-SAE features, using raw feature activations instead of weighting byIS⁡\(f\)\\mathrm\{IS\}\(f\)\. This avoids mechanically suppressing the lower\-validation features that the audit is designed to inspect\.

We then rank all active interpretation features by their mass underBallB^\{\\mathrm\{all\}\}for each topic, and focus on lower\-validation features withIS⁡\(f\)<\.8\\mathrm\{IS\}\(f\)<\.8that would have ranked among the top 20 features for at least one topic\. These features are the strongest possible challenge to the validated descriptor vocabulary: they are highly topic\-associated, but absent from the final MonoTM representation\. For each such feature, assigned to the topic where it obtains its best rank, we search for a validated same\-topic neighbor among the top\-50 validated features of that topic underBallB^\{\\mathrm\{all\}\}\. Appendix[L](https://arxiv.org/html/2609.09575#A12)gives the full construction, row alignment, EM objective, filtering rules, and formal definitions\.

Coverage is measured in document\-activation space\. For each excluded feature, we compute cosine similarity between its raw document\-level interpretation\-SAE activation vector and the corresponding vectors for validated same\-topic descriptors\. We compare the nearest validated same\-topic descriptor to a random validated descriptor sampled from the same topic\-specific candidate pool\. Figure[1](https://arxiv.org/html/2609.09575#S5.F1)shows that nearest validated descriptors are much closer than random validated descriptors, especially for 20 Newsgroups and Web of Science\.

Figure 1:Coverage of lower\-validation high\-association features by validated same\-topic descriptors\. Bars show the median cosine similarity between feature activation profiles across documents\.The Reuters results are less straightforward, so we further characterize the excluded high\-association features using heuristic label\-type flags based on keywords in the generated feature labels\. \(See Appendix[L](https://arxiv.org/html/2609.09575#A12)for the keyword rules\.\) These flags are used to summarize broad tendencies in the excluded feature set, not as a validation metric\.

Table[2](https://arxiv.org/html/2609.09575#S5.T2)shows a clear dataset difference\. In 20 Newsgroups and Web of Science, excluded high\-association features are mostly semantic or broad\-discourse labels\. In Reuters, by contrast, 70\.7% are flagged as format/register/numeric cues, suggesting that many of these features capture financial\-news register, reporting format, or numeric conventions rather than missing core topic semantics\. Concrete examples of both patterns are provided in Appendix[L](https://arxiv.org/html/2609.09575#A12)\.

DatasetSemanticBroaddiscourseFormat/register/numeric20NG58\.1%36\.4%5\.4%WoS88\.5%7\.7%3\.8%Reuters6\.3%23\.1%70\.7%Table 2:Heuristic label\-type split for lower\-validation high\-association features\. Percentages are computed within each dataset\.Overall, while this audit does not prove that no excluded feature ever captures a meaningful omitted semantic dimension, it alleviates the concern that MonoTM arbitrarily omits a large class of independent topic\-defining features\.

#### 5\.3\.3Descriptor Comparison

Finally, we compare descriptor mechanisms under the same fixed document–topic mixturesΘ\\Theta\. This comparison is intended to isolate the descriptor layer by asking how different mechanisms describe the same inferred topics\. We compare MonoTM against three descriptor baselines:θ\\theta\-weighted TF\-IDF terms, BERTopic\-style c\-TF\-IDF terms\([Grootendorst, 2022](https://arxiv.org/html/2609.09575#bib.bib2)\), and free\-form LLM\-generated descriptors\. We include the LLM baseline because recent work increasingly treats LLMs as topic extractors\([Lam et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib39);[Mu et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib38);[Pham et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib37)\)\. Full descriptor\-generation details, prompts, and topic\-name compression settings are provided in Appendix[M](https://arxiv.org/html/2609.09575#A13)\. Full topic\-name comparison tables are provided in Appendix[O](https://arxiv.org/html/2609.09575#A15), and representative descriptor lists before topic\-name compression are provided in Appendix[P](https://arxiv.org/html/2609.09575#A16)\.

The comparison suggests two qualitative patterns\. First, word\-based descriptors often recover salient lexical anchors but remain surface\-level\. For example, in 20 Newsgroups, for the topic aligned withtalk\.politics\.misc, the word\-based baselines produce the topic name “Gun rights,” whereas MonoTM yields “American culture wars,” reflecting a broader mixture of gun\-rights, sexuality, morality, and civil\-liberties discussions \(Table[12](https://arxiv.org/html/2609.09575#A15.T12)\)\. Second, free\-form LLM descriptors can be fluent but overly local to the sampled topic\-associated documents\. In Reuters, for example, the LLM baseline names the topic aligned withcrudeas “Ecuador oil crisis,” whereas MonoTM yields the broader “OPEC oil market” \(Table[14](https://arxiv.org/html/2609.09575#A15.T14)\)\. These examples suggest that MonoTM descriptors offer a useful intermediate level of semantic abstraction: they bridge high\-level topic labels and raw documents more effectively than word lists, while remaining more directly tied to the representations used for topic inference than ad hoc LLM descriptors\.

Beyond topic naming, MonoTM also supports downstream analyses of relations among topics\. Appendix[Q](https://arxiv.org/html/2609.09575#A17)illustrates this use case by comparing topic relations in the validated\-feature space\.

## 6Conclusion

We introducedMonoTM, an interpretable topic modeling framework built on sparse autoencoders trained over document embeddings\. Across three research questions, our results show that SAE features are useful for topic modeling in multiple but distinct ways\. First, corpus\-trained SAEs can produce relatively monosemantic units, but the availability of reliably interpretable features depends strongly on the choice of SAE configuration\. Second, SAE activations can also serve as effective statistical units for estimating document–topic mixtures\. However, the SAE configurations suited to these two roles do not necessarily coincide\. Addressing RQ3, MonoTM resolves this mismatch by estimating document–topic mixtures from the full SAE representation and then learning topic–feature distributions over validated features with the mixtures fixed\. Our analyses further show that excluded high\-association features are typically covered by validated same\-topic descriptors or reflect auxiliary non\-core cues\.

## 7Limitations

##### Remaining uncertainty about excluded features\.

Section[5\.3\.2](https://arxiv.org/html/2609.09575#S5.SS3.SSS2)audits features from the interpretation SAE that fall below the validation threshold and finds that genuinely hard\-to\-validate features account for only a small share of high\-association topic\-feature mass\. However, this audit does not prove that all excluded features are unimportant or fully understood\. Some low\-validation features may encode corpus\-specific regularities, pragmatic cues, formatting patterns, or culturally specific concepts that are difficult to summarize with short semantic labels\. Moreover, our validation scores depend on the Interpreter and Predictor LLMs, so systematic blind spots in those models could still affect which features are included in the final descriptor set\. For high\-impact applications, MonoTM should therefore be paired with additional coverage audits and, when appropriate, human expert review\.

##### Sensitivity to the Interpreter and Predictor LLMs\.

Our interpretability scores depend on the capabilities and failure modes of the specific LLMs used as Interpreter and Predictor, as well as prompt details and decoding settings\. Different LLM backends may vary in \(i\) their ability to abstract from examples into stable hypotheses, \(ii\) calibration on the binary prediction task, and \(iii\) robustness to domain\-specific language\. As a result, both the number of features that validate above a threshold and the apparent granularity of validated features could change with the choice of model\(s\)\. While we partially mitigate this by separating generation \(Interpreter\) from validation \(Predictor\), our validation remains an LLM\-mediated measurement rather than a ground\-truth guarantee of monosemanticity\.

##### Cost and practicality of LLM\-heavy autointerpretability\.

MonoTM’s feature labeling relies on many LLM calls, which is computationally expensive and can be slow in practice\. This cost is a real barrier for iterative modeling workflows\. That said, MonoTM is aimed at settings where researchers are willing to spend more compute to obtain a single, carefully audited interpretive lens on an important corpus \(e\.g\., social science and digital humanities analyses where interpretability and traceability are first\-order goals\)\. Still, improving efficiency is crucial\. Promising directions include cheaper feature screening models for “likely interpretable” features and distilling the Predictor into a small classifier\. Additionally, continuing progress in SAE feature interpretation and tooling\([Lieberum et al\., 2024](https://arxiv.org/html/2609.09575#bib.bib32), e\.g\.,\)suggests that MonoTM could become cheaper to apply while preserving its core advantages\.

##### Systematic biases can distort the interpretable topic representation\.

MonoTM estimates document–topic mixtures using the full SAE vocabulary but represents topics using only the subset of features that pass LLM\-based validation\. If the Interpreter or Predictor has systematic blind spots—for example, consistently under\-interpreting certain registers, dialects, domains, or culturally specific concepts—then those features may be disproportionately excluded from the interpretable feature set\. In that case, the topic descriptors produced by MonoTM could omit important aspects of a topic even if those aspects strongly influence the inferred mixtures\. This limitation is especially salient if the LLMs’ biases correlate with sensitive attributes or with particular styles of expression in the corpus\. Mitigations include auditing interpretability coverage across document subpopulations, validating with multiple LLM backends, and incorporating human expert review for high\-impact analyses\.

## References

- Babakhinet al\.\(2025\)Y\. Babakhin, R\. Osmulski, R\. Ak, G\. Moreira, M\. Xu, B\. Schifferer, B\. Liu, and E\. OldridgeLlama\-embed\-nemotron\-8b: a universal text embedding model for multilingual and cross\-lingual tasks\.External Links:2511\.07025,[Link](https://arxiv.org/abs/2511.07025)Cited by:[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px2.p1.1)\.
- Bernhard\-Harreret al\.\(2025\)J\. Bernhard\-Harrer, R\. Ashour, J\. Eberl, P\. Tolochko, and H\. BoomgaardenBeyond standardization: a comprehensive review of topic modeling validation methods for computational social science research\.Political Science Research and Methods,pp\. 1–19\(en\)\.External Links:ISSN 2049\-8470, 2049\-8489,[Link](https://www.cambridge.org/core/journals/political-science-research-and-methods/article/beyond-standardization-a-comprehensive-review-of-topic-modeling-validation-methods-for-computational-social-science-research/1D92CCD7C51491C9F6BE490BA5B434C4),[Document](https://dx.doi.org/10.1017/psrm.2025.10008)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p1.1)\.
- Bianchiet al\.\(2021a\)F\. Bianchi, S\. Terragni, D\. Hovy, D\. Nozza, and E\. FersiniCross\-lingual Contextualized Topic Models with Zero\-shot Learning\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,P\. Merlo, J\. Tiedemann, and R\. Tsarfaty \(Eds\.\),Online,pp\. 1676–1683\.External Links:[Link](https://aclanthology.org/2021.eacl-main.143/),[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.143)Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Bianchiet al\.\(2021b\)F\. Bianchi, S\. Terragni, and D\. HovyPre\-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 759–766\.External Links:[Link](https://aclanthology.org/2021.acl-short.96/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-short.96)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Billset al\.\(2023\)S\. Bills, N\. Cammarata, D\. Mossing, H\. Tillman, L\. Gao, G\. Goh, I\. Sutskever, J\. Leike, J\. Wu, and W\. SaundersLanguage models can explain neurons in language models\.OpenAI\(en\)\.External Links:[Link](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html)Cited by:[§2\.1](https://arxiv.org/html/2609.09575#S2.SS1.p2.1),[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px4.p1.1)\.
- Bleiet al\.\(2003\)D\. M\. Blei, A\. Y\. Ng, and M\. I\. JordanLatent Dirichlet Allocation\.Journal of Machine Learning Research3,pp\. 993–1022\.Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Brickenet al\.\(2023\)T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. OlahTowards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2023/monosemantic\-features/index\.htmlExternal Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features/index.html)Cited by:[§2\.1](https://arxiv.org/html/2609.09575#S2.SS1.p2.1)\.
- Changet al\.\(2009\)J\. Chang, S\. Gerrish, C\. Wang, J\. Boyd\-graber, and D\. BleiReading Tea Leaves: How Humans Interpret Topic Models\.InAdvances in Neural Information Processing Systems,Vol\.22\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2009/hash/f92586a25bb3145facd64ab20fd554ff-Abstract.html)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p2.1)\.
- Da \(2019\)N\. Z\. DaThe Computational Case against Computational Literary Studies\.Critical Inquiry45\(3\),pp\. 601–639\.Note:Publisher: The University of Chicago PressExternal Links:ISSN 0093\-1896,[Link](https://www.jstor.org/stable/26621899)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p1.1)\.
- Dienget al\.\(2020\)A\. B\. Dieng, F\. J\. R\. Ruiz, and D\. M\. BleiTopic Modeling in Embedding Spaces\.Transactions of the Association for Computational Linguistics8,pp\. 439–453\.Note:Place: Cambridge, MA Publisher: MIT PressExternal Links:[Link](https://aclanthology.org/2020.tacl-1.29/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00325)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px3.p1.1)\.
- Elhageet al\.\(2022\)N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen, R\. Grosse, S\. McCandlish, J\. Kaplan, D\. Amodei, M\. Wattenberg, and C\. OlahToy models of superposition\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/toy_model/index.html)Cited by:[§2\.1](https://arxiv.org/html/2609.09575#S2.SS1.p1.1)\.
- Girrbach and Akata \(2025\)L\. Girrbach and Z\. AkataSparse Autoencoders are Topic Models\.arXiv\.Note:arXiv:2511\.16309 \[cs\]External Links:[Link](http://arxiv.org/abs/2511.16309),[Document](https://dx.doi.org/10.48550/arXiv.2511.16309)Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px2.p1.1)\.
- Grimmer and Stewart \(2013\)J\. Grimmer and B\. M\. StewartText as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts\.Political Analysis21\(3\),pp\. 267–297\(en\)\.External Links:ISSN 1047\-1987, 1476\-4989,[Link](https://www.cambridge.org/core/journals/political-analysis/article/text-as-data-the-promise-and-pitfalls-of-automatic-content-analysis-methods-for-political-texts/F7AAC8B2909441603FEB25C156448F20),[Document](https://dx.doi.org/10.1093/pan/mps028)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p1.1)\.
- Grootendorst \(2022\)M\. GrootendorstBERTopic: Neural topic modeling with a class\-based TF\-IDF procedure\.arXiv\.Note:arXiv:2203\.05794 \[cs\]External Links:[Link](http://arxiv.org/abs/2203.05794),[Document](https://dx.doi.org/10.48550/arXiv.2203.05794)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p2.1),[§5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3.p1.1)\.
- Hubenet al\.\(2024\)R\. Huben, H\. Cunningham, L\. R\. Smith, A\. Ewart, and L\. SharkeySparse Autoencoders Find Highly Interpretable Features in Language Models\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=F76bwRSLeK)Cited by:[§2\.1](https://arxiv.org/html/2609.09575#S2.SS1.p2.1)\.
- Kardoset al\.\(2025\)M\. Kardos, J\. Kostkan, K\. Enevoldsen, A\. Vermillet, K\. Nielbo, and R\. RoccaS3\{\}^\{3\}– Semantic Signal Separation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 633–666\.External Links:ISBN 979\-8\-89176\-251\-0,[Link](https://aclanthology.org/2025.acl-long.32/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.32)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Kowsariet al\.\(2017\)K\. Kowsari, D\. E\. Brown, M\. Heidarysafa, K\. Jafari Meimandi, M\. S\. Gerber, and L\. E\. BarnesHDLTex: hierarchical deep learning for text classification\.InMachine Learning and Applications \(ICMLA\), 2017 16th IEEE International Conference on,Cited by:[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px1.p1.1)\.
- Lamet al\.\(2024\)M\. S\. Lam, J\. Teoh, J\. A\. Landay, J\. Heer, and M\. S\. BernsteinConcept Induction: Analyzing Unstructured Text with High\-Level Concepts Using LLooM\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,CHI ’24,New York, NY, USA,pp\. 1–28\.External Links:ISBN 979\-8\-4007\-0330\-0,[Link](https://dl.acm.org/doi/10.1145/3613904.3642830),[Document](https://dx.doi.org/10.1145/3613904.3642830)Cited by:[§5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3.p1.1)\.
- Lang \(1995\)K\. LangNewsWeeder: learning to filter netnews\.InProceedings of the Twelfth International Conference on Machine Learning,Cited by:[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px1.p1.1)\.
- Lauet al\.\(2014\)J\. H\. Lau, D\. Newman, and T\. BaldwinMachine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality\.InProceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics,S\. Wintner, S\. Goldwater, and S\. Riezler \(Eds\.\),Gothenburg, Sweden,pp\. 530–539\.External Links:[Link](https://aclanthology.org/E14-1056),[Document](https://dx.doi.org/10.3115/v1/E14-1056)Cited by:[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px3.p1.1)\.
- Lewis \(1997\)D\. D\. LewisReuters\-21578 text categorization test collection\.Distribution 1\.0\.Cited by:[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px1.p1.1)\.
- Lhoestet al\.\(2021\)Q\. Lhoest, A\. Villanova del Moral, Y\. Jernite, A\. Thakur, P\. von Platen, S\. Patil, J\. Chaumond, M\. Drame, J\. Plu, L\. Tunstall, J\. Davison, M\. Šaško, G\. Chhablani, B\. Malik, S\. Brandeis, T\. Le Scao, V\. Sanh, C\. Xu, N\. Patry, A\. McMillan\-Major, P\. Schmid, S\. Gugger, C\. Delangue, T\. Matussière, L\. Debut, S\. Bekman, P\. Cistac, T\. Goehringer, V\. Mustar, F\. Lagunas, A\. Rush, and T\. WolfDatasets: A Community Library for Natural Language Processing\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,H\. Adel and S\. Shi \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 175–184\.External Links:[Link](https://aclanthology.org/2021.emnlp-demo.21/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-demo.21)Cited by:[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px1.p1.1)\.
- Lieberumet al\.\(2024\)T\. Lieberum, S\. Rajamanoharan, A\. Conmy, L\. Smith, N\. Sonnerat, V\. Varma, J\. Kramar, A\. Dragan, R\. Shah, and N\. NandaGemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, N\. Kim, J\. Jumelet, H\. Mohebbi, A\. Mueller, and H\. Chen \(Eds\.\),Miami, Florida, US,pp\. 278–300\.External Links:[Link](https://aclanthology.org/2024.blackboxnlp-1.19/),[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.19)Cited by:[§7](https://arxiv.org/html/2609.09575#S7.SS0.SSS0.Px3.p1.1)\.
- Meiet al\.\(2007\)Q\. Mei, X\. Shen, and C\. ZhaiAutomatic labeling of multinomial topic models\.InProceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining,San Jose California USA,pp\. 490–499\(en\)\.External Links:ISBN 978\-1\-59593\-609\-7,[Link](https://dl.acm.org/doi/10.1145/1281192.1281246),[Document](https://dx.doi.org/10.1145/1281192.1281246)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p2.1)\.
- Miaoet al\.\(2017\)Y\. Miao, E\. Grefenstette, and P\. BlunsomDiscovering Discrete Latent Topics with Neural Variational Inference\.InProceedings of the 34th International Conference on Machine Learning,pp\. 2410–2419\(en\)\.Note:ISSN: 2640\-3498External Links:[Link](https://proceedings.mlr.press/v70/miao17a.html)Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p1.1)\.
- Mimnoet al\.\(2011\)D\. Mimno, H\. Wallach, E\. Talley, M\. Leenders, and A\. McCallumOptimizing Semantic Coherence in Topic Models\.InProceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,R\. Barzilay and M\. Johnson \(Eds\.\),Edinburgh, Scotland, UK\.,pp\. 262–272\.External Links:[Link](https://aclanthology.org/D11-1024)Cited by:[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px3.p1.1)\.
- Muet al\.\(2024\)Y\. Mu, C\. Dong, K\. Bontcheva, and X\. SongLarge Language Models Offer an Alternative to the Traditional Approach of Topic Modelling\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 10160–10171\.External Links:[Link](https://aclanthology.org/2024.lrec-main.887/)Cited by:[§5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3.p1.1)\.
- Newmanet al\.\(2009\)D\. Newman, S\. Karimi, and L\. CavedonExternal evaluation of topic models\.InProceedings of the 14th Australasian Document Computing Symposium,pp\. 1–8\.Cited by:[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px3.p1.1)\.
- NVIDIA \(2025\)NVIDIANvidia/llama\-embed\-nemotron\-8b \(hugging face model card/readme\)\.Note:[https://huggingface\.co/nvidia/llama\-embed\-nemotron\-8b](https://huggingface.co/nvidia/llama-embed-nemotron-8b)Accessed 2025\-12\-31Cited by:[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px2.p1.1)\.
- Olahet al\.\(2020\)C\. Olah, N\. Cammarata, L\. Schubert, G\. Goh, M\. Petrov, and S\. CarterZoom in: an introduction to circuits\.Distill\.Note:https://distill\.pub/2020/circuits/zoom\-inExternal Links:[Document](https://dx.doi.org/10.23915/distill.00024.001),[Link](https://distill.pub/2020/circuits/zoom-in)Cited by:[§2\.1](https://arxiv.org/html/2609.09575#S2.SS1.p1.1)\.
- O’Neillet al\.\(2024\)C\. O’Neill, C\. Ye, K\. G\. Iyer, and J\. F\. WuTowards Interpretable Scientific Foundation Models: Sparse Autoencoders for Disentangling Dense Embeddings of Scientific Concepts\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=mPq3R6jdtD)Cited by:[§2\.1](https://arxiv.org/html/2609.09575#S2.SS1.p2.1),[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.09575#S4.SS0.SSS0.Px4.p1.1)\.
- Phamet al\.\(2024\)C\. M\. Pham, A\. Hoyle, S\. Sun, P\. Resnik, and M\. IyyerTopicGPT: A Prompt\-based Topic Modeling Framework\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2956–2984\.External Links:[Link](https://aclanthology.org/2024.naacl-long.164/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.164)Cited by:[§5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3.p1.1)\.
- Röderet al\.\(2015\)M\. Röder, A\. Both, and A\. HinneburgExploring the Space of Topic Coherence Measures\.InProceedings of the Eighth ACM International Conference on Web Search and Data Mining,WSDM ’15,New York, NY, USA,pp\. 399–408\.External Links:ISBN 978\-1\-4503\-3317\-7,[Link](https://dl.acm.org/doi/10.1145/2684822.2685324),[Document](https://dx.doi.org/10.1145/2684822.2685324)Cited by:[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px3.p1.1)\.
- Shadrova \(2021\)A\. ShadrovaTopic models do not model topics: epistemological remarks and steps towards best practices\.Journal of Data Mining & Digital Humanities2021\(en\)\.Note:Publisher: Episciences\.orgExternal Links:ISSN 2416\-5999,[Link](https://jdmdh.episciences.org/8608),[Document](https://dx.doi.org/10.46298/jdmdh.7595)Cited by:[§1](https://arxiv.org/html/2609.09575#S1.p1.1)\.
- Srivastava and Sutton \(2017\)A\. Srivastava and C\. SuttonAutoencoding Variational Inference For Topic Models\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=BybtVK9lg)Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p1.1)\.
- Wuet al\.\(2023\)X\. Wu, X\. Dong, T\. T\. Nguyen, and A\. T\. LuuEffective Neural Topic Modeling with Embedding Clustering Regularization\.InProceedings of the 40th International Conference on Machine Learning,pp\. 37335–37357\(en\)\.Note:ISSN: 2640\-3498External Links:[Link](https://proceedings.mlr.press/v202/wu23c.html)Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024a\)X\. Wu, T\. Nguyen, and A\. T\. LuuA survey on neural topic models: methods, applications, and challenges\.Artificial Intelligence Review57\(2\),pp\. 18\(en\)\.External Links:ISSN 1573\-7462,[Link](https://doi.org/10.1007/s10462-023-10661-7),[Document](https://dx.doi.org/10.1007/s10462-023-10661-7)Cited by:[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024b\)X\. Wu, T\. Nguyen, D\. C\. Zhang, W\. Y\. Wang, and A\. T\. LuuFASTopic: pretrained transformer is a fast, adaptive, stable, and transferable topic model\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24, Vol\.37,Red Hook, NY, USA,pp\. 84447–84481\.External Links:ISBN 979\-8\-3313\-1438\-5Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, N\. Beltran\-Velez, S\. Karlekar, C\. Shi, A\. Nazaret, A\. Mallik, A\. Feder, and D\. M\. BleiModel Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders\.arXiv\.Note:arXiv:2507\.23220 \[cs\]External Links:[Link](http://arxiv.org/abs/2507.23220),[Document](https://dx.doi.org/10.48550/arXiv.2507.23220)Cited by:[§2\.2](https://arxiv.org/html/2609.09575#S2.SS2.SSS0.Px2.p1.1)\.

## Appendix ADataset Details

See Table[3](https://arxiv.org/html/2609.09575#A1.T3)for summary statistics of the datasets\.

DatasetPointsCatsMultiNo LabelMean CharsSD CharsMedianMinMax20 Newsgroups1884620X01902\.533984\.971175115160616Web of Science469857X01376\.46492\.771355958019Reuters1256147O2304843\.7874\.975465613396Table 3:Summary statistics of the datasets used\. The abbreviations in the header are defined as follows: Points \(Total number of data points\), Cats \(Number of categories\), and Multi \(Multi\-label classification, where ’O’ denotes multi\-label and ’X’ denotes single\-label\)\. No Label indicates the number of samples without a label\. The character count statistics \(Mean, SD, Median, Min, Max\) describe the distribution of document lengths\.
## Appendix BSAE Architecture and Training Details

For each documentd∈𝒟=\{1,…,M\}d\\in\\mathcal\{D\}=\\\{1,\\dots,M\\\}, letxd∈ℝmx\_\{d\}\\in\\mathbb\{R\}^\{m\}denote its document embedding and letx~d∈ℝm\\tilde\{x\}\_\{d\}\\in\\mathbb\{R\}^\{m\}denote the dimension\-wise standardized embedding, using the corpus\-level mean and standard deviation\. In our experiments,m=4096m=4096\.

We use a single\-layer top\-KKsparse autoencoder with hidden dimensionalitym​NmN, whereNNis the expansion factor\. Depending onNN, the dictionary may be undercomplete, complete, or overcomplete relative to the embedding dimension\. The decoder is a matrixD∈ℝm×\(m​N\)D\\in\\mathbb\{R\}^\{m\\times\(mN\)\}, and the encoder uses tied weightsD⊤D^\{\\top\}:

z\\displaystyle z=D⊤​x~\+b,\\displaystyle=D^\{\\top\}\\tilde\{x\}\+b,\(1\)a\\displaystyle a=ReLU⁡\(z\)\.\\displaystyle=\\mathrm\{ReLU\}\(z\)\.\(2\)Sparsity is enforced with a hard top\-KKoperator,

h=TopK⁡\(a,K\),h=\\mathrm\{TopK\}\(a,K\),\(3\)which yields a nonnegative latent activation vectorh∈ℝ≥0m​Nh\\in\\mathbb\{R\}^\{mN\}\_\{\\geq 0\}with at mostKKactive coordinates\.

We additionally introduce a learnable per\-feature gain vectorg∈ℝ\>0m​Ng\\in\\mathbb\{R\}^\{mN\}\_\{\>0\}to modulate feature magnitudes independently of decoder direction norms\. We parameterizeggin log\-space to enforce positivity and define

h~=h⊙g\.\\tilde\{h\}=h\\odot g\.\(4\)The reconstruction of the standardized embedding is

x^=D​h~\.\\hat\{x\}=D\\tilde\{h\}\.\(5\)After each update, decoder columns are normalized to unit norm to avoid arbitrary rescaling between decoder weights and latent activations\.

We train each SAE with Adam using learning rate10−410^\{\-4\}and batch size 1024 for up to 300k steps\. We select the checkpoint with the lowest normalized reconstruction error on a fixed monitoring split containing 10% of documents\. The training objective is

ℒ=‖x~−x^‖22‖x~−x¯‖22\+ε,\\mathcal\{L\}=\\frac\{\\\|\\tilde\{x\}\-\\hat\{x\}\\\|\_\{2\}^\{2\}\}\{\\\|\\tilde\{x\}\-\\bar\{x\}\\\|\_\{2\}^\{2\}\+\\varepsilon\},\(6\)wherex¯\\bar\{x\}is the batch mean\.

We train a grid of top\-KKSAEs over\(N,K\)\(N,K\):

N\\displaystyle N∈\{0\.015625,0\.03125,0\.0625,0\.125,\\displaystyle\\in\\\{0\.015625,0\.03125,0\.0625,0\.125,0\.25,0\.5,1,2,3,4,5\},\\displaystyle 0\.25,0\.5,1,2,3,4,5\\\},K\\displaystyle K∈\{4,8,16,32,64,128,256,512\}\.\\displaystyle\\in\\\{4,8,16,32,64,128,256,512\\\}\.This gives up to11×8=8811\\times 8=88configurations\. We omit configurations withm​N<KmN<K, since they cannot activateKKdistinct features\. Whenm​N=KmN=K, the top\-KKoperator retains all units, so the model reduces to a dense tied\-weight autoencoder; we keep this boundary case as a diagnostic comparison\.

## Appendix CPrompts for the Interpreter–Predictor Module

See Figure[2](https://arxiv.org/html/2609.09575#A3.F2)for the Interpreter prompt and the Predictor prompt\.

Interpreter PromptYou are an NLP researcher conducting an investigation into one common basis vector shared by several document embeddings\. Your goal is to determine what semantic phenomenon this vector encodes—specifically, which linguistic cue, tone, word choice, narrative framing, concept, or topic is associated with high activation along this dimension\.INPUT DESCRIPTION: You will be given three inputs:1\.Max\-Activating Examples — up to 15 texts that produce high activation on this feature, with activation scores\.2\.Typical\-Activating Examples — up to 15 texts that activate this feature but are not necessarily among the highest activations\.3\.Zero\-Activating Examples — up to 15 texts that produce zero activation on this feature\.OUTPUT DESCRIPTION: Using the inputs, complete the following:1\.From theMax\-Activating Examples, list potential topics, concepts, themes, frames, and linguistic features they share\. Consider multiple granularities\. Give greater weight to items more prominent in higher\-activation examples\.2\.Cross\-check those candidates against theTypical\-Activating Examples\. Keep features that generalize across both max and typical positives; de\-emphasize items that only appear in extreme cases\.3\.Using theZero\-Activating Examples, rule out any surviving topics/concepts/features that also appear in the zero\-activating set\. Be systematic\.4\.Based on steps 1–3, provide a concise, rational analysis of the cues, tone, wording, frames, concepts, or topics—at appropriate granularity—that are likely to activate this neuron\. Prefer the simplest explanation that fits all evidence\.5\.Pick the single most prominent component that aligns with this vector, and express it in 4–10 words as:FINAL:<description\>\.Here are the max\-activating examples:<max activating examples\>
Here are the typical\-activating examples:<typical activating examples\>
Here are the zero\-activating examples:<zero activating examples\>Work through the steps thoroughly and analytically to interpret the vector\. Output should only be in this format:FINAL:<description\>\. Do NOT return anything after these 4–10 words\.Predictor PromptYou are an NLP researcher investigating one of the basis vectors of a document\-embedding space\. Your goal is to predict whether a given text exhibits the feature encoded by this vector\.OUTPUT DESCRIPTION: Based on the feature description, predict whether the vector will activate for this text\. If you predict activation, output 1\. If not, output 0\.Here is the description of the feature encoded by the vector:<description\>
Here is the text to predict:<text\>Provide the output only in one of the following forms:PREDICTION: 1orPREDICTION: 0\. Do not include anything else\.
Figure 2:The prompts used for the Interpreter–Predictor Module\. The Interpreter generates a semantic description of the vector, which is then fed into the Predictor to validate predictive accuracy\.
## Appendix DLLM Models and Inference Settings

For Interpreter and Predictor LLM runs, we use open\-weight, instruction\-tuned Gemma 3 models: Gemma\-3\-27B\-IT \(FP8\) as the Interpreter and Gemma\-3\-12B\-IT \(bfloat16\) as the Predictor\.222[https://huggingface\.co/google/gemma\-3\-27b\-it](https://huggingface.co/google/gemma-3-27b-it)and[https://huggingface\.co/google/gemma\-3\-12b\-it](https://huggingface.co/google/gemma-3-12b-it)\.Both models are accessed via the DeepInfra API\.333[https://deepinfra\.com](https://deepinfra.com/)\.We use the smaller model for the Predictor because the task is a comparatively simple binary decision\. For both Interpreter and Predictor calls, we set the maximum context length to 131,072 tokens\.

## Appendix ECounts of Interpretable Features Across Interpretability Score Thresholds

See Table[4](https://arxiv.org/html/2609.09575#A5.T4), Table[5](https://arxiv.org/html/2609.09575#A5.T5)and Table[6](https://arxiv.org/html/2609.09575#A5.T6)\.

NNKKIS≥0\.95\\geq 0\.95IS≥0\.90\\geq 0\.90IS≥0\.85\\geq 0\.85IS≥0\.80\\geq 0\.800\.54732393744531489211339423241032173123723487193280328448318426130254951992723070\.5853237483721186726950268928622374135473846184320434486115927435758491422503220\.516792292577116221614819232162221856410193162417544372141628163326476516491943655120\.5321228428213254520760423298337599333233278851175943273462108117955329154012321923Table 4:Number of features above interpretability score thresholds in the 20 Newsgroups dataset\. IS denotes the interpretability score\.NNKKIS≥0\.95\\geq 0\.95IS≥0\.90\\geq 0\.90IS≥0\.85\\geq 0\.85IS≥0\.80\\geq 0\.800\.54224470587686142765687418582438375995810633443883210361169444678611086118854509936112112540\.5824764893411221837992412941555285941231167419833865513591816213048709144319102195586571439192422040\.51610546686312221163431037173323312165741734257332743167041900280635364166871899283535675166641826285535510\.532211784227731329254311991924232293130125323857332403168331694656432443180834355120532376172433605082Table 5:Number of features above interpretability score thresholds in the Web of Science dataset\. IS denotes the interpretability score\.NNKKIS≥0\.95\\geq 0\.95IS≥0\.90\\geq 0\.90IS≥0\.85\\geq 0\.85IS≥0\.80\\geq 0\.800\.5459152213277145012018722624581221802103442108169204444611616319654471081581900\.58391232634061843152270372283812820227938421051842614832931662365831781431920\.5162110223843311624125266467216211062063473163110219328441642146251358516551682824000\.532330121285132866220515232331774467563325726957088143258272584983532552896541090Table 6:Number of features above interpretability score thresholds in the Reuters dataset\. IS denotes the interpretability score\.
## Appendix FFeature Granularity Across SAE Configurations

### F\.1Motivation

In the main text, RQ1 focuses on whether corpus\-trained SAE features can be reliably interpreted as semantic units\. As a secondary diagnostic, we also ask whether SAE configurations differ in the granularity of the validated semantic features they recover\. If larger or more active SAEs refine features learned by smaller SAEs, then document\-level activations from the larger SAE should more easily reconstruct the validated\-feature activations of the smaller SAE than vice versa\.

This analysis is not required for MonoTM, but it helps characterize how SAE configuration affects the structure of the validated feature space\.

### F\.2Probe setup

Consider two SAEs trained on the same corpus but with different configurations\(N,K\)\(N,K\), denoted A and B\. Letℱ\(A\)\\mathcal\{F\}^\{\(A\)\}andℱ\(B\)\\mathcal\{F\}^\{\(B\)\}be the subsets of latent features that pass our label validation threshold\. We useIS≥0\.80\\mathrm\{IS\}\\geq 0\.80throughout\. Letp=\|ℱ\(A\)\|p=\|\\mathcal\{F\}^\{\(A\)\}\|andq=\|ℱ\(B\)\|q=\|\\mathcal\{F\}^\{\(B\)\}\|, and letMMbe the number of documents\.

### F\.3Document–feature activation matrices

For each documentdd, each SAE produces a sparse top\-KKactivation list\{\(i,ad​i\)\}i∈TopK⁡\(d\)\\\{\(i,a\_\{di\}\)\\\}\_\{i\\in\\mathrm\{TopK\}\(d\)\}withad​i≥0a\_\{di\}\\geq 0\. We construct a dense document–feature activation vector by accumulating these latent activations into the coordinates corresponding to validated features\. For SAE A, we definexd\(A\)∈ℝ≥0px^\{\(A\)\}\_\{d\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{p\}by

xd​j\(A\)=∑i∈TopK\(A\)​\(d\)ad​i\(A\)\[i=fj\(A\)\],x^\{\(A\)\}\_\{dj\}=\\sum\_\{i\\in\\mathrm\{TopK\}^\{\(A\)\}\(d\)\}a^\{\(A\)\}\_\{di\}\\,\\mathbf\{1\}\\\!\\left\[i=f^\{\(A\)\}\_\{j\}\\right\],\(7\)where\(f1\(A\),…,fp\(A\)\)\(f^\{\(A\)\}\_\{1\},\\dots,f^\{\(A\)\}\_\{p\}\)is a fixed ordering ofℱ\(A\)\\mathcal\{F\}^\{\(A\)\}\. Equivalently, we keep only validated feature coordinates and set all others to zero\. Stacking over documents yieldsX\(A\)∈ℝ≥0M×pX^\{\(A\)\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{M\\times p\}\. We analogously constructX\(B\)∈ℝ≥0M×qX^\{\(B\)\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{M\\times q\}\.

### F\.4Linear mapping probe

We fit an affine linear map in each direction:

X^\(B\)\\displaystyle\\widehat\{X\}^\{\(B\)\}=X\(A\)​WA→B⊤\+𝟏​bA→B⊤,\\displaystyle=X^\{\(A\)\}W\_\{A\\to B\}^\{\\top\}\+\\mathbf\{1\}\\,b\_\{A\\to B\}^\{\\top\},\(8\)X^\(A\)\\displaystyle\\widehat\{X\}^\{\(A\)\}=X\(B\)​WB→A⊤\+𝟏​bB→A⊤,\\displaystyle=X^\{\(B\)\}W\_\{B\\to A\}^\{\\top\}\+\\mathbf\{1\}\\,b\_\{B\\to A\}^\{\\top\},\(9\)whereWA→B∈ℝq×pW\_\{A\\to B\}\\in\\mathbb\{R\}^\{q\\times p\}andbA→B∈ℝqb\_\{A\\to B\}\\in\\mathbb\{R\}^\{q\}, and symmetrically forB→AB\\to A\. Parameters are estimated on a training split using ridge regression\.444Since our goal is to quantify reconstructability rather than minimize training error, we use ridge regression to control coefficient magnitudes and reduce sensitivity to resampling, yielding a lower\-variance reconstructability estimate\.

### F\.5Error metric and directional asymmetry

Because document–feature activations are nonnegative and sparse, we evaluate reconstruction with a weighted MSE that assigns higher weights to nonzero target entries:

WMSE⁡\(Y^,Y\)=∑d,jwd​j​\(y^d​j−yd​j\)2∑d,jwd​j,wd​j=1\+\(λ−1\)1\[yd​j\>0\],\\mathrm\{WMSE\}\(\\widehat\{Y\},Y\)=\\frac\{\\sum\_\{d,j\}w\_\{dj\}\(\\widehat\{y\}\_\{dj\}\-y\_\{dj\}\)^\{2\}\}\{\\sum\_\{d,j\}w\_\{dj\}\},\\\\ w\_\{dj\}=1\+\(\\lambda\-1\)\\,\\mathbf\{1\}\[y\_\{dj\}\>0\],\(10\)withλ=10\\lambda=10in our experiments\.

To make scores comparable across different target spaces, we normalize by the WMSE of a zero\-predictor baseline computed on the same validation set:

ρ⁡\(A→B\)=WMSE⁡\(X^\(B\),X\(B\)\)WMSE⁡\(0,X\(B\)\)\.\\rho\(A\\to B\)=\\frac\{\\mathrm\{WMSE\}\(\\widehat\{X\}^\{\(B\)\},X^\{\(B\)\}\)\}\{\\mathrm\{WMSE\}\(0,X^\{\(B\)\}\)\}\.\(11\)Lowerρ\\rhoindicates more accurate reconstruction relative to predicting all\-zero activations\.

Our main statistic is the directional asymmetry

Δ=ρ⁡\(big→small\)−ρ⁡\(small→big\),\\Delta=\\rho\(\\text\{big\}\\to\\text\{small\}\)\-\\rho\(\\text\{small\}\\to\\text\{big\}\),\(12\)where “big” and “small” refer to the ordered SAE configurations used in this probe\. Negative values indicate that the larger SAE more easily reconstructs the validated\-feature activation patterns of the smaller SAE than the reverse direction\.

### F\.6Resampling protocol

For each dataset, we repeat the full procedure over 10 random train/validation splits of documents\. For each split, we fit both directions,A→BA\\to BandB→AB\\to A, and computeρ\\rhoon the held\-out validation documents\. We evaluate all ordered pairs among the four SAE configurations shown in Figure[3](https://arxiv.org/html/2609.09575#A6.F3)\.

### F\.7Results

DatasetMedianΔ\\Delta95% boot\. CIPr⁡\(Δ<0\)\\Pr\(\\Delta<0\)20 Ng−0\.111\-0\.111\[−0\.223,0\.052\]\[\-0\.223,\\;\\;0\.052\]0\.6670\.667WoS−0\.099\-0\.099\[−0\.164,−0\.048\]\[\-0\.164,\\;\-0\.048\]1\.0001\.000Reuters−0\.102\-0\.102\[−0\.139,−0\.057\]\[\-0\.139,\\;\-0\.057\]1\.0001\.000Table 7:Summary of cross\-SAE predictability asymmetry across datasets\. Reported are the medianΔ\\Deltaand a 95% hierarchical bootstrap confidence interval over configuration pairs and random splits, along withPr⁡\(Δ<0\)\\Pr\(\\Delta<0\)\.![Refer to caption](https://arxiv.org/html/2609.09575v1/rho_heatmap_grid_mean_sd.png)Figure 3:Heatmaps of linear cross\-SAE reconstruction\. Each cell shows the mean validation MSE ratioρ\\rhofor predicting the target SAE activations from the source SAE activations, with direction shown as row→\\rightarrowcolumn\. Parentheses give the standard deviation over 10 train/validation splits\. Lower is better\.Figure[3](https://arxiv.org/html/2609.09575#A6.F3)shows directional asymmetries in cross\-SAE predictability\. On Web of Science and Reuters, mappings from higher\-capacity configurations to lower\-capacity ones consistently achieve lower normalized error than the reverse direction across essentially all configuration pairs\. Aggregating across all pairs and seeds, the medianΔ\\Deltais negative on both datasets \(Table[7](https://arxiv.org/html/2609.09575#A6.T7)\), indicating that larger SAEs more easily reconstruct smaller SAEs than vice versa under this linear probe\.

For 20 Newsgroups, the aggregate medianΔ\\Deltais also negative but less conclusive under resampling\. The asymmetry is strongest when comparing the most separated configurations,\(N=2,K=4\)\(N\{=\}2,K\{=\}4\)and\(N=5,K=32\)\(N\{=\}5,K\{=\}32\), consistent with a larger gap in representational resolution\.

### F\.8Interpretation and limitations

Taken together, these results are consistent with a granularity shift as SAE capacity and activity increase\. Features learned in larger SAEs appear to contain sufficient information to linearly reconstruct the validated\-feature activations of smaller SAEs, while the reverse reconstruction is systematically harder\.

This probe does not establish one\-to\-one feature correspondences, and it may underestimate nonlinear relationships between feature spaces\. It should therefore be interpreted as a diagnostic of representational resolution rather than as direct evidence that individual features split cleanly across SAE configurations\.

## Appendix GDocument–Topic Mixture Evaluation Details

We evaluate document–topic mixture quality through alignment with gold label topics\. This evaluation is intended to measure whether the inferred topic mixtures recover the benchmark category structure, rather than whether topic descriptors form coherent word lists\.

For 20 Newsgroups and Web of Science, which are single\-label datasets, we assign each document to its highest\-probability inferred topic,

z^d=arg⁡maxk⁡θd,k\.\\hat\{z\}\_\{d\}=\\arg\\max\_\{k\}\\theta\_\{d,k\}\.We then compute the confusion matrix between inferred topics and gold labels and use the Hungarian algorithm to find the one\-to\-one topic–label matching that maximizes total agreement\. After applying this mapping, we treat the aligned topic assignments as multi\-class predictions and report Micro\-F1 and Macro\-F1\.

Reuters is multi\-label and highly imbalanced, so we evaluate it as a multi\-label topic–label alignment problem\. For each candidate threshold

t∈\{0\.1,0\.2,0\.3,0\.4,0\.5\},t\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\},we predict topickkfor documentddwhenθd,k\>t\\theta\_\{d,k\}\>t\. We then compute pairwise F1 scores between each gold label topic and each inferred topic, use the Hungarian algorithm to obtain a one\-to\-one label–topic matching, and compute Micro\-F1 and Macro\-F1 over the matched pairs\. We apply the same threshold grid to all models and report each metric at its best threshold over this grid\. Documents with no Reuters gold label are retained in the evaluation, so a model can avoid false positives on these documents only by assigning all topic probabilities below the selected threshold\.

## Appendix HFull SAE Hyperparameter Heatmaps for Mixture Estimation

Figure[4](https://arxiv.org/html/2609.09575#A8.F4)reports Micro\-F1 and Macro\-F1 over the full SAE hyperparameter grid for BoF\+LDA on each dataset\. Figure[5](https://arxiv.org/html/2609.09575#A8.F5)provides a complementary aggregate view by averaging performance across datasets after scaling each dataset’s F1 scores to the\[0,1\]\[0,1\]range\. Together, these figures support the hyperparameter\-selection discussion in Section[5\.2\.1](https://arxiv.org/html/2609.09575#S5.SS2.SSS1)by showing how document–topic mixture quality varies with dictionary capacityNNand per\-document activityKK\.

Across datasets, two patterns are visible\. First, very wide SAEs paired with very smallKKtend to perform poorly\. In this regime, the dictionary has high capacity, but each document is allowed to activate only a small number of features\. The resulting BoF representation is therefore too constrained: it exposes many possible pseudo\-token types, but provides too little per\-document evidence for stable topic inference\. Second, boundary cases wherem​N=KmN=Kare also weak\. Since the top\-KKoperator retains all available units in this case, the model no longer imposes a meaningful sparsity bottleneck, which appears to reduce the usefulness of SAE latents as topic\-modeling signals\.

By contrast, stronger performance appears in a balanced scaling band whereNNandKKincrease together\. These configurations preserve an effective sparsity constraint while allowing each document to express a richer set of SAE features\. This pattern is visible in the per\-dataset heatmaps in Figure[4](https://arxiv.org/html/2609.09575#A8.F4)and remains apparent in the dataset\-aggregated heatmaps in Figure[5](https://arxiv.org/html/2609.09575#A8.F5)\. The purple region in Figure[4](https://arxiv.org/html/2609.09575#A8.F4)marks the robust subset of this balanced regime used to summarize BoF\+LDA in Table[1](https://arxiv.org/html/2609.09575#S5.T1)\. We use this region rather than a single best cell to avoid overemphasizing dataset\- or seed\-specific hyperparameter choices\.

![Refer to caption](https://arxiv.org/html/2609.09575v1/rq2_heatmaps_highlighted.png)Figure 4:Heatmaps of Micro\-F1 and Macro\-F1 scores over the full SAE hyperparameter grid for BoF\+LDA\. The purple box indicates the\(N,K\)\(N,K\)region whose values are averaged to produce the scores reported in Table[1](https://arxiv.org/html/2609.09575#S5.T1)\.![Refer to caption](https://arxiv.org/html/2609.09575v1/scaled_heatmaps.png)Figure 5:Heatmaps of Micro\-F1 \(top\) and Macro\-F1 \(bottom\) scores, shown as mean \(standard deviation\) over three datasets, computed from 9 data points for each\(N,K\)\(N,K\)\. F1 scores were scaled to the\[0,1\]\[0,1\]range on a per\-dataset basis to prevent any single dataset from dominating performance\.
## Appendix IDocument–Topic Estimation Using Only Interpretable Features

See Figure[6](https://arxiv.org/html/2609.09575#A9.F6), Figure[7](https://arxiv.org/html/2609.09575#A9.F7), and Figure[8](https://arxiv.org/html/2609.09575#A9.F8)\.

![Refer to caption](https://arxiv.org/html/2609.09575v1/20NewsGroup_20topics_heatmaps_4x2_F5_6_7_8_3.png)Figure 6:Heatmaps of micro\-F1 and macro\-F1 scores for document–topic mixtures on the 20 Newsgroups dataset, inferred using only features with interpretability scores \(IS\) above various cutoffs\.![Refer to caption](https://arxiv.org/html/2609.09575v1/webofscience_7topics_heatmaps_4x2_F5_6_7_8.png)Figure 7:Heatmaps of micro\-F1 and macro\-F1 scores for document–topic mixtures on the Web of Science dataset, inferred using only features with interpretability scores \(IS\) above various cutoffs\.![Refer to caption](https://arxiv.org/html/2609.09575v1/reuters_47topics_thr0p2_heatmaps_4x2_F5_6_7_8.png)Figure 8:Heatmaps of micro\-F1 and macro\-F1 scores for document–topic mixtures on the Reuters dataset, inferred using only features with interpretability scores \(IS\) above various cutoffs\. To evaluate against the multi\-label gold labels, we treat topics whose inferred document–topic probabilities exceed 0\.2 as predicted classes\.
## Appendix JSample Topic–Feature Representations

Table[8](https://arxiv.org/html/2609.09575#A10.T8)presents a subset of MonoTM’s topic representation output, reporting the top\-5 labeled features per topic for three topics in each dataset\.

We use an interpretable\-feature SAE with\(N=1,K=8\)\(N\{=\}1,K\{=\}8\)and filter features byIS⁡\(f\)≥0\.8\\mathrm\{IS\}\(f\)\\geq 0\.8\. This is not the richest configuration we tested: larger settings, such as\(N=5,K=32\)\(N\{=\}5,K\{=\}32\), yield substantially more interpretable features\. However, the top feature labels for each topic show that even with interpretable features obtained from a relatively small SAE, the feature labels associated with highβ\\betavalues align well with the gold topic labels and provide a finer\-grained understanding of the corpus than topic labels alone\.

DatasetTopicFeature labelβ\\beta20 Newsgroupsrec\.sport\.hockeyNHL hockey game and player information\.40220 Newsgroupsrec\.sport\.hockeySports scores reported on Usenet newsgroups\.04820 Newsgroupsrec\.sport\.hockeyFrustrated fan complaints about sports broadcasting decisions\.03820 Newsgroupsrec\.sport\.hockeyPassionate sports discussion in online forums\.03120 Newsgroupsrec\.sport\.hockeySports stats requests and early internet culture\.03020 Newsgroupssoc\.religion\.christianIntense Christian theological debate and interpretation\.34820 Newsgroupssoc\.religion\.christianReligious debate, interpretation, and personal belief defense\.06120 Newsgroupssoc\.religion\.christianTraditional Catholic theology and liturgical concerns\.04120 Newsgroupssoc\.religion\.christianSkepticism towards religious belief and dogma\.04020 Newsgroupssoc\.religion\.christianChristian theological discussion and apologetic reasoning\.03020 Newsgroupsrec\.motorcyclesMotorcycle riding advice and safety discussion\.48120 Newsgroupsrec\.motorcyclesMotorcycle countersteering technique and related debate\.03520 Newsgroupsrec\.motorcyclesDiscussions of drunk driving and motorcycles\.03320 Newsgroupsrec\.motorcyclesMotorcycle culture, camaraderie, road\-based social signaling\.03220 Newsgroupsrec\.motorcyclesMotorcycle enthusiast discussion: ownership, specs, and banter\.031Web of ScienceMedicalCardiovascular disease research and clinical information\.016Web of ScienceMedicalGastrointestinal function, microbiome, and symptomology\.016Web of ScienceMedicalBladder dysfunction, urological assessment, and clinical research\.015Web of ScienceMedicalOsteoporosis research, bone density, and skeletal health\.014Web of ScienceMedicalNeurobiological basis of psychiatric disease research\.013Web of SciencePsychologyInternal psychological states and research analysis\.029Web of SciencePsychologyParenting challenges, family stress, child well\-being\.025Web of SciencePsychologySocial judgment and character attribution processes\.020Web of SciencePsychologyPsychiatric comorbidity, distress, and vulnerability factors\.019Web of SciencePsychologyAutism spectrum disorder and related neurodevelopmental features\.017Web of ScienceCSAdvanced computer and network security threat analysis\.030Web of ScienceCSComputer vision algorithm development and evaluation\.023Web of ScienceCSParallel computing, performance optimization, technical implementation details\.022Web of ScienceCSCryptographic security and data privacy techniques\.019Web of ScienceCSMachine learning methodology and technical rigor\.016ReutersearnComparative financial performance reporting, earnings vs prior period\.516ReutersearnComparative financial performance, “X vs Y” reporting\.175ReutersearnQuarterly financial report key metrics comparison\.073ReutersearnFinancial turnaround, profit versus loss reporting\.063ReutersearnCorporate quarterly financial results reporting and metrics\.054ReutersacqCorporate acquisition and ownership transfer announcements\.311ReutersacqCorporate mergers and acquisitions announcements, financial details\.089ReutersacqMergers, acquisitions, and corporate restructuring events\.088ReutersacqCorporate merger and acquisition announcements \(LOI\)\.081ReutersacqEnergy sector financial deal reporting details\.051Reutersmoney\-fxCentral bank money market interventions and rates\.366Reutersmoney\-fxBank of England money market liquidity revisions\.227Reutersmoney\-fxCentral bank money market intervention reporting\.097Reutersmoney\-fxBank of England liquidity intervention reporting\.075Reutersmoney\-fxBritish monetary policy and currency market signals\.033Table 8:Part of the final MonoTM output showing topic–feature distributions\. The table lists the top five validated feature labels for three topics from each dataset\. Gold labels from the original datasets are assigned using the alignment procedure in Section[5\.2](https://arxiv.org/html/2609.09575#S5.SS2)and are shown only for orientation\.β\\betadenotes the probability of each feature within the topic–feature distribution\.
## Appendix KFull MonoTM Algorithm and Inference Details

This appendix expands the MonoTM algorithm summarized in Section[5\.3\.1](https://arxiv.org/html/2609.09575#S5.SS3.SSS1)\. Let𝒟=\{1,…,D\}\\mathcal\{D\}=\\\{1,\\dots,D\\\}index documents and letTTbe the number of topics\. MonoTM estimates document–topic mixtures from a full SAE bag\-of\-features representation, then estimates a topic–feature distribution over validated interpretable SAE features with the mixtures held fixed\.

We allow the SAE used for mixture estimation and the SAE used for interpretation to be different\. We denote their activations byh~mix\\tilde\{h\}^\{\\mathrm\{mix\}\}andh~int\\tilde\{h\}^\{\\mathrm\{int\}\}, respectively\. The single\-SAE case is recovered by setting these two representations to be the same\. For each documentdd, letθd∈ΔT\\theta\_\{d\}\\in\\Delta^\{T\}denote its document–topic mixture, and letΘ∈ℝ≥0D×T\\Theta\\in\\mathbb\{R\}\_\{\\geq 0\}^\{D\\times T\}collect these mixtures row\-wise\. Letℱτ=\{1,…,V\}\\mathcal\{F\}\_\{\\tau\}=\\\{1,\\dots,V\\\}denote the set of validated interpretable SAE features used for topic description\. Each featuref∈ℱτf\\in\\mathcal\{F\}\_\{\\tau\}has a natural\-language labelℓf\\ell\_\{f\}and interpretability scoreIS⁡\(f\)\\mathrm\{IS\}\(f\)\. Our goal is to estimate a topic–feature matrixB∈ℝ≥0T×VB\\in\\mathbb\{R\}\_\{\\geq 0\}^\{T\\times V\}whose rowβt∈ΔV\\beta\_\{t\}\\in\\Delta^\{V\}ranks validated interpretable features for topictt\.

##### Stage 1: estimating document–topic mixtures\.

For mixture estimation, MonoTM uses the full mixture\-SAE representation rather than only validated features\. For a top\-KKSAE with expansion factorNN, each document embedding yields a sparse nonnegative activation vectorh~dmix∈ℝ≥0m​N\\tilde\{h\}^\{\\mathrm\{mix\}\}\_\{d\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{mN\}with at mostKKnonzero entries\. We construct a document–feature matrixXmix∈ℝ≥0D×m​NX^\{\\mathrm\{mix\}\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{D\\times mN\}by setting

Xd,jmix=h~d,jmix,X^\{\\mathrm\{mix\}\}\_\{d,j\}=\\tilde\{h\}^\{\\mathrm\{mix\}\}\_\{d,j\},\(13\)and fit LDA withTTtopics onXmixX^\{\\mathrm\{mix\}\}to obtainΘ\\Theta\. This stage uses all active SAE features because Section[5\.2\.2](https://arxiv.org/html/2609.09575#S5.SS2.SSS2)shows that strict interpretability filtering generally hurts mixture estimation\.

##### Stage 2: constructing the interpretable feature matrix\.

For topic description, MonoTM restricts to interpretation\-SAE features that \(i\) activate in at least 30 documents, \(ii\) have a nonempty natural\-language label, and \(iii\) satisfyIS⁡\(f\)≥τ\\mathrm\{IS\}\(f\)\\geq\\tau, withτ=0\.8\\tau=0\.8in our experiments\. The interpretable document–feature matrixC∈ℝ≥0D×VC\\in\\mathbb\{R\}\_\{\\geq 0\}^\{D\\times V\}is defined by

cd,f=max⁡\(h~d,fint,0\)⋅IS⁡\(f\),f∈ℱτ\.c\_\{d,f\}=\\max\(\\tilde\{h\}^\{\\mathrm\{int\}\}\_\{d,f\},0\)\\cdot\\mathrm\{IS\}\(f\),\\qquad f\\in\\mathcal\{F\}\_\{\\tau\}\.\(14\)This weighting makes the final topic descriptors emphasize features with stronger validation evidence\.

##### Stage 3: estimating topic–feature distributions\.

GivenΘ\\ThetaandCC, MonoTM estimatesBBby maximizing the fixed\-Θ\\Thetamixture log\-likelihood

ℒ⁡\(B\)=∑d=1D∑f=1Vcd,f​log⁡\(∑t=1Tθd,t​βt,f\),\\mathcal\{L\}\(B\)=\\sum\_\{d=1\}^\{D\}\\sum\_\{f=1\}^\{V\}c\_\{d,f\}\\log\\Big\(\\sum\_\{t=1\}^\{T\}\\theta\_\{d,t\}\\beta\_\{t,f\}\\Big\),\(15\)subject toβt∈ΔV\\beta\_\{t\}\\in\\Delta^\{V\}for alltt\. After estimatingBB, topicttis represented by its ranked feature distributionβt\\beta\_\{t\}; the reported descriptor list consists of the top\-ranked labelsℓf\\ell\_\{f\}underβt,f\\beta\_\{t,f\}, with associated probabilities\.

### K\.1Dirichlet\-smoothed EM

We optimize Eq\. \([15](https://arxiv.org/html/2609.09575#A11.E15)\) with an EM algorithm that treats latent topic assignments as missing data\. Let

qd,f,t=p⁡\(z=t∣d,f\)=θd,t​βt,f∑t′θd,t′​βt′,f\.q\_\{d,f,t\}=p\(z=t\\mid d,f\)=\\frac\{\\theta\_\{d,t\}\\beta\_\{t,f\}\}\{\\sum\_\{t^\{\\prime\}\}\\theta\_\{d,t^\{\\prime\}\}\\beta\_\{t^\{\\prime\},f\}\}\.\(16\)The E\-step computes expected topic–feature counts:

Nt,f=∑d=1Dcd,f​qd,f,t\.N\_\{t,f\}=\\sum\_\{d=1\}^\{D\}c\_\{d,f\}\\,q\_\{d,f,t\}\.\(17\)The M\-step updatesBBwith symmetric Dirichlet smoothingη\>0\\eta\>0:

βt,f=Nt,f\+η∑f′\(Nt,f′\+η\)\.\\beta\_\{t,f\}=\\frac\{N\_\{t,f\}\+\\eta\}\{\\sum\_\{f^\{\\prime\}\}\(N\_\{t,f^\{\\prime\}\}\+\\eta\)\}\.\(18\)We iterate until relative improvement in Eq\. \([15](https://arxiv.org/html/2609.09575#A11.E15)\) falls below a tolerance\. To avoid zero\-probability features and stabilize topics whenCCis sparse, we use the data\-dependent heuristic

η=1T\+L¯V,\\eta=\\frac\{1\}\{T\}\+\\frac\{\\bar\{L\}\}\{V\},\(19\)whereL¯\\bar\{L\}is the average number of nonzero validated features per document\.

### K\.2Using Separate Mixture and Interpretation SAEs

MonoTM does not require a single SAE to simultaneously optimize mixture estimation and interpretability\. Even when a single SAE is used end\-to\-end, we do not interpret topics using the topic–feature distribution implicitly produced duringΘ\\Thetainference\. Instead, topic interpretation is based on a separately estimated topic–feature distribution over validated interpretable features\. When the SAE configuration that yields the most useful validated feature set is suboptimal for estimatingΘ\\Theta, a user can train a second SAE optimized for mixture quality while using an interpretability\-optimized SAE to constructCC\.

## Appendix LExcluded\-feature audit details

This appendix gives the full procedure for the excluded\-feature audit in Section[5\.3\.2](https://arxiv.org/html/2609.09575#S5.SS3.SSS2)\. The audit asks whether features excluded from the final MonoTM descriptor vocabulary nevertheless contain topic\-defining semantic information that is absent from the validated descriptor set\.

##### Configurations\.

For all datasets, the document–topic mixture matrixΘ\\Thetais the same one used in the main RQ3 analysis\. It is estimated from the mixture\-quality SAE with expansion factorN=0\.5N=0\.5and top\-K=256K=256, using seed 123\. The number of topics is fixed to the number of gold categories:T=20T=20for 20 Newsgroups,T=7T=7for Web of Science, andT=47T=47for Reuters\. The interpretation\-side SAE used for the audit hasN=2N=2andK=16K=16\.

##### All\-active interpretation feature matrix\.

Letℱact\\mathcal\{F\}\_\{\\mathrm\{act\}\}denote the set of interpretation\-SAE features that activate in at least 30 documents, and letℓf\\ell\_\{f\}be the generated label for featureff\. The validated descriptor set used by MonoTM is

ℱval=\{f∈ℱact:IS⁡\(f\)≥\.8​and​ℓf≠∅\}\.\\mathcal\{F\}\_\{\\mathrm\{val\}\}=\\\{f\\in\\mathcal\{F\}\_\{\\mathrm\{act\}\}:\\mathrm\{IS\}\(f\)\\geq\.8\\text\{ and \}\\ell\_\{f\}\\neq\\emptyset\\\}\.For the audit, we instead construct an all\-active feature matrixA∈ℝ≥0D×\|ℱact\|A\\in\\mathbb\{R\}\_\{\\geq 0\}^\{D\\times\|\\mathcal\{F\}\_\{\\mathrm\{act\}\}\|\}with entries

ad,f=max⁡\(h~d,f,0\),a\_\{d,f\}=\\max\(\\tilde\{h\}\_\{d,f\},0\),whereh~d,f\\tilde\{h\}\_\{d,f\}is the gain\-adjusted top\-KKactivation of the interpretation SAE\. Unlike the final descriptor estimation step, the audit does not weight feature activations byIS⁡\(f\)\\mathrm\{IS\}\(f\)\. This avoids building the conclusion into the measurement: interpretability\-score weighting would mechanically reduce the mass of lower\-validation features\. Document rows are aligned betweenΘ\\Theta, the interpretation\-SAE activations, and raw documents using saved row IDs\.

##### All\-feature audit distribution\.

Using fixedΘ\\Thetaand the raw all\-active matrixAA, we estimate an auxiliary topic–feature distributionBallB^\{\\mathrm\{all\}\}by maximizing

ℒ⁡\(Ball\)=∑d=1D∑f∈ℱactad,f​log⁡\(∑t=1Tθd,t​βt,fall\),\\mathcal\{L\}\(B^\{\\mathrm\{all\}\}\)=\\sum\_\{d=1\}^\{D\}\\sum\_\{f\\in\\mathcal\{F\}\_\{\\mathrm\{act\}\}\}a\_\{d,f\}\\log\\left\(\\sum\_\{t=1\}^\{T\}\\theta\_\{d,t\}\\beta^\{\\mathrm\{all\}\}\_\{t,f\}\\right\),subject toβtall∈Δ\|ℱact\|\\beta^\{\\mathrm\{all\}\}\_\{t\}\\in\\Delta^\{\|\\mathcal\{F\}\_\{\\mathrm\{act\}\}\|\}for every topictt\. We use the same Dirichlet\-smoothed EM routine as in the fixed\-Θ\\ThetaMonoTM descriptor step\. This distribution is used only for auditing; the final MonoTM descriptors are still estimated over the validated feature vocabulary\.

##### High\-risk excluded set\.

For each topictt, all active features are ranked byβt,fall\\beta^\{\\mathrm\{all\}\}\_\{t,f\}\. We define

ranktall⁡\(f\)=1\+\|\{f′:βt,f′all\>βt,fall\}\|\.\\operatorname\{rank\}^\{\\mathrm\{all\}\}\_\{t\}\(f\)=1\+\\left\|\\\{f^\{\\prime\}:\\beta^\{\\mathrm\{all\}\}\_\{t,f^\{\\prime\}\}\>\\beta^\{\\mathrm\{all\}\}\_\{t,f\}\\\}\\right\|\.For each featureffwithIS⁡\(f\)<\.8\\mathrm\{IS\}\(f\)<\.8, lettf⋆t\_\{f\}^\{\\star\}be the topic where it has its best rank\. The high\-risk excluded set is

ℛ20=\{\(tf⋆,f\):IS\(f\)<\.8,ranktf⋆all\(f\)≤20\}\.\\mathcal\{R\}\_\{20\}=\\\{\(t\_\{f\}^\{\\star\},f\):\\mathrm\{IS\}\(f\)<\.8,\\;\\operatorname\{rank\}^\{\\mathrm\{all\}\}\_\{t\_\{f\}^\{\\star\}\}\(f\)\\leq 20\\\}\.This set contains lower\-validation features that are sufficiently associated with at least one topic to be plausible missing descriptors\.

##### Coverage test\.

For each\(t,f\)∈ℛ20\(t,f\)\\in\\mathcal\{R\}\_\{20\}, we compareffwith validated same\-topic descriptors\. The candidate pool is

𝒱t,M=\{g∈ℱval:ranktall⁡\(g\)≤M\}\.\\mathcal\{V\}\_\{t,M\}=\\\{g\\in\\mathcal\{F\}\_\{\\mathrm\{val\}\}:\\operatorname\{rank\}^\{\\mathrm\{all\}\}\_\{t\}\(g\)\\leq M\\\}\.We reportM=50M=50in the main text and includeM=20M=20andM=100M=100as sensitivity settings\. TheM=20M=20setting asks whether the feature is covered by the most compact validated descriptor region;M=50M=50asks whether it is covered by the broader high\-ranked validated descriptor distribution;M=100M=100checks sensitivity to a wider candidate set\.

Letafa\_\{f\}be the document\-level activation vector for featureff\. We compute

cosact⁡\(f,g\)=af⊤​ag‖af‖2​‖ag‖2\.\\cos\_\{\\mathrm\{act\}\}\(f,g\)=\\frac\{a\_\{f\}^\{\\top\}a\_\{g\}\}\{\\\|a\_\{f\}\\\|\_\{2\}\\\|a\_\{g\}\\\|\_\{2\}\}\.The nearest validated same\-topic descriptor is

g⋆​\(f,t\)=arg⁡maxg∈𝒱t,M​cosact⁡\(f,g\)\.g^\{\\star\}\(f,t\)=\\arg\\max\_\{g\\in\\mathcal\{V\}\_\{t,M\}\}\\cos\_\{\\mathrm\{act\}\}\(f,g\)\.For comparison, we sample random validated descriptors from the same candidate pool and compute the random\-baseline activation cosine\. Table[9](https://arxiv.org/html/2609.09575#A12.T9)reports the aggregate results\.

DatasetCandidate poolnnMed\. nearestMed\. randomMed\.Δ\\Delta%cosact≥\.2\\%\\cos\_\{\\mathrm\{act\}\}\\geq\.220NGtop\-20129\.239\.072\.15460\.520NGtop\-50129\.295\.055\.21872\.920NGtop\-100129\.312\.044\.26278\.3WoStop\-2026\.360\.104\.23576\.9WoStop\-5026\.419\.079\.29588\.5WoStop\-10026\.442\.066\.34996\.2Reuterstop\-20334\.170\.059\.10541\.6Reuterstop\-50334\.170\.043\.13045\.2Reuterstop\-100334\.188\.029\.15347\.9Table 9:Coverage audit details for lower\-validation high\-association features\. The high\-risk excluded set is fixed as features withIS⁡\(f\)<\.8\\mathrm\{IS\}\(f\)<\.8and best all\-feature topic rank at most 20\. Candidate pool size controls how many validated same\-topic descriptors are allowed to provide coverage\. Median random cosine is computed from random validated descriptors sampled from the same topic\-specific candidate pool\.
##### Heuristic label\-type flags\.

We also assign a coarse heuristic label type to each excluded feature based only on keywords in its generated feature label\. These flags are not manual annotations and are not used as a validation metric\. They are included to characterize broad patterns, especially the difference between Reuters and the other datasets\.

Labels are flagged asformat/register/numericif they include terms related to document form, boilerplate, reporting style, technical artifacts, numerical data, or financial\-news announcements, including keywords such as:format, boilerplate, template, placeholder, metadata, header, posting, email, message, announcement, reporting, reports, quarterly, qtly, dividend, earnings, record date, subscription, unsubscribe, FAQ, manual, instructions, technical support, code, script, program, software package, release, version, archive, file, conversion, table, tabular, numerical, numeric, quantitative, percentage, percent, price, market data, Reuter, newswire\. Labels are flagged asbroad discourseif they include terms related to debate, critique, controversy, policy, institutional or political discussion, economic conflict, or broad framing, including keywords such as:discussion, debate, critique, controversy, tensions, policy, institutional, economic, financial, trade, political, rhetoric, argument, concerns, issues, context, general, broad\. Remaining labels are assigned the defaultsemanticflag\. The rules are applied in this order: format/register/numeric first, broad discourse second, and semantic as the default\.

##### Examples from the excluded\-feature audit\.

Table[10](https://arxiv.org/html/2609.09575#A12.T10)gives deterministically selected examples from the label\-type audit\. To avoid cherry\-picking, examples are selected by a fixed rule: within each dataset and heuristic label\-type category, we sort features by best all\-feature topic rank, break ties by largerβt,fall\\beta^\{\\mathrm\{all\}\}\_\{t,f\}, and show the first available feature\. The table is intended to make the heuristic categories transparent, not to serve as a separate evaluation\.

DatasetTypeIS⁡\(f\)\\mathrm\{IS\}\(f\)Excluded labelNearest validated labelcosact\\cos\_\{\\mathrm\{act\}\}20NGsemantic\.410Patient narratives of challenging medical experiencesAlternative medicine debate and medical authority skepticism\.33720NGbroad\.783Detailed, analytical discussion of professional hockeyToronto Maple Leafs playoff game discussion & fandom\.30120NGformat/reg\.\.727Sports news and scores reporting via UsenetIronic, irreverent, and self\-aware internet discourse\.176WoSsemantic\.333Rainwater harvesting for water resource managementRainwater harvesting for sustainable water management\.732WoSbroad\.750Distributed computing system architecture and design discussIntelligent interconnected devices and adaptive environments\.443WoSformat/reg\.\.750Novel materials for photocatalytic energy conversionPhotocatalysis, solar energy, material science research\.692Reuterssemantic\.667Bank ownership stake acquisitions and controlCorporate deals, mergers, and joint ventures reporting\.341Reutersbroad\.769International trade disputes and policy tensionsInternational trade disputes and government intervention\.315Reutersformat/reg\.\.256US\-USSR grain trade reporting, quantitative dataU\.S\. farm policy and commodity support programs\.312Table 10:Deterministically selected examples from the excluded\-feature label\-type audit\. Examples are selected by sorting, within each dataset and heuristic type, by best all\-feature topic rank and then by larger audit\-distribution topic mass\.Concrete high\-association cases illustrate the two main patterns\. In 20 Newsgroups, an excluded feature labeled “Israeli policy critique with Nazi analogies” is ranked first underBallB^\{\\mathrm\{all\}\}for the topic aligned totalk\.politics\.mideast\. Its nearest validated same\-topic descriptor is “Israeli\-Palestinian conflict, criticism, and political dispute,” with activation cosine\.650\.650\. In Web of Science, the excluded feature “Rainwater harvesting for water resource management” is ranked first for the Civil topic, but its nearest validated same\-topic descriptor is “Rainwater harvesting for sustainable water management,” with activation cosine\.732\.732\. These cases look less like independent missing topics and more like lower\-validation variants of semantic regions already covered by validated descriptors\. By contrast, Reuters features such as “Financial dividend and earnings report announcements” or “Dividend and earnings financial report language” are systematic and topic\-associated, but they primarily describe reporting format or financial\-news register rather than substantive topical content\.

## Appendix MDescriptor Baseline and Topic\-Naming Details

For the controlled descriptor comparison in Section[5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3), all methods describe the same fixed document–topic mixturesΘ\\Theta\. We generate descriptors for each inferred topic using four mechanisms:θ\\theta\-weighted TF\-IDF terms, BERTopic\-style c\-TF\-IDF terms, free\-form LLM\-generated descriptors, and MonoTM validated SAE\-feature descriptors\.

##### Word\-based descriptor preprocessing\.

Both word\-based baselines are implemented withscikit\-learnvectorizers\. We do not apply stemming or lemmatization\. The vectorizers lowercase all text, remove English stopwords, and use unigrams only\. We use the token pattern

\(?u\)\\b\[a\-zA\-Z\]\[a\-zA\-Z0\-9\_\\\-\]\{2,\}\\b,\\texttt\{\(?u\)\\textbackslash b\[a\-zA\-Z\]\[a\-zA\-Z0\-9\\\_\\textbackslash\-\]\\lx@text@lbrace 2,\\lx@text@rbrace\\textbackslash b\},which keeps tokens that begin with an ASCII letter and have length at least three, while allowing later alphanumeric, underscore, and hyphen characters\. This removes many short tokens, numbers\-only tokens, and punctuation artifacts\. We use a maximum vocabulary size of 50,000 terms\.

##### θ\\theta\-weighted TF\-IDF\.

For theθ\\theta\-weighted TF\-IDF baseline, we fit aTfidfVectorizeron the full aligned corpus withlowercase=True,stop\_words=english,ngram\_range=\(1,1\),min\_df=2,max\_df=0\.95,max\_features=50000,norm=None, andsublinear\_tf=True\. For each topictt, letDt50D\_\{t\}^\{50\}be the 50 documents with the largestθd,t\\theta\_\{d,t\}\. We normalize topic weights within this top\-document set,

θ¯d,t=θd,t∑d′∈Dt50θd′,t,\\bar\{\\theta\}\_\{d,t\}=\\frac\{\\theta\_\{d,t\}\}\{\\sum\_\{d^\{\\prime\}\\in D\_\{t\}^\{50\}\}\\theta\_\{d^\{\\prime\},t\}\},and score each termwwby

sttfidf​\(w\)=∑d∈Dt50θ¯d,t​tfidfd,w\.s\_\{t\}^\{\\mathrm\{tfidf\}\}\(w\)=\\sum\_\{d\\in D\_\{t\}^\{50\}\}\\bar\{\\theta\}\_\{d,t\}\\operatorname\{tfidf\}\_\{d,w\}\.We report the 30 highest\-scoring terms for each topic\.

##### c\-TF\-IDF\.

For the c\-TF\-IDF baseline, we form one class document for each topic by concatenating the same 50 top\-θ\\thetadocuments\. We then fit aCountVectorizerwithlowercase=True,stop\_words=english,ngram\_range=\(1,1\),min\_df=1,max\_features=50000, and the same token pattern as above\. Letnt,wn\_\{t,w\}be the count of termwwin topic class documenttt, and letLt=∑wnt,wL\_\{t\}=\\sum\_\{w\}n\_\{t,w\}be the class\-document length\. We compute

tft,w=nt,wLt\.\\operatorname\{tf\}\_\{t,w\}=\\frac\{n\_\{t,w\}\}\{L\_\{t\}\}\.Letdfclass⁡\(w\)\\operatorname\{df\}\_\{\\mathrm\{class\}\}\(w\)be the number of topic class documents in whichwwappears, and letL¯\\bar\{L\}be the average class\-document length across topics\. We use the class\-based inverse\-document\-frequency term

idfw=log⁡\(1\+L¯max⁡\(dfclass⁡\(w\),1\)\),\\operatorname\{idf\}\_\{w\}=\\log\\left\(1\+\\frac\{\\bar\{L\}\}\{\\max\(\\operatorname\{df\}\_\{\\mathrm\{class\}\}\(w\),1\)\}\\right\),and rank terms by

stctfidf​\(w\)=tft,w⁡idfw\.s\_\{t\}^\{\\mathrm\{ctfidf\}\}\(w\)=\\operatorname\{tf\}\_\{t,w\}\\operatorname\{idf\}\_\{w\}\.We report the 30 highest\-scoring terms for each topic\.

##### Free\-form LLM descriptor baseline\.

The LLM summary baseline is intended to be a strong post\-hoc descriptor baseline\. For each topic, the prompt receives 50 topic\-associated documents\. The 25 highest\-θ\\thetadocuments are always included, and the remaining 25 documents are sampled without replacement from the top 200 documents for that topic using a fixed random seed\. Each document is first whitespace\-normalized and then truncated to at most 1000 model tokens; if tokenization support is unavailable, the implementation falls back to whitespace\-token truncation\. We do not apply an additional character cap\.

We use OpenAIgpt\-5\-2025\-08\-07through the Responses API\. For LLM\-generated descriptors, we usereasoning\_effort=low,verbosity=low, JSON\-object output mode, and a maximum output budget of 3000 tokens\. We leave temperature unspecified\. If the response cannot be parsed into exactly 10 descriptors, we retry up to three times\. The prompt asks the model to produce exactly 10 concise, non\-redundant descriptors grounded only in the provided excerpts\.

Free\-form LLM descriptor promptYou are given excerpts from documents that are highly associated with the same latent topic\.
Task: Produce exactly 10 concise descriptors for this topic\.
Rules: Each descriptor must be 4 to 10 words\. Descriptors should be specific, non\-redundant, and grounded only in the excerpts\. Do not output a single topic name; output a ranked list of descriptors\. Return JSON only, with this exact schema:\{"descriptors": \["\.\.\.", "\.\.\."\]\}\.
Topic id:<topic\>
Documents:<document excerpts\>
##### MonoTM descriptors\.

For MonoTM, we use the topic–feature distributionBBestimated by the fixed\-Θ\\Thetadescriptor model described in Appendix[K](https://arxiv.org/html/2609.09575#A11)\. We retain validated SAE features withIS⁡\(f\)≥0\.8\\mathrm\{IS\}\(f\)\\geq 0\.8and report the top 10 feature labels underβt,f\\beta\_\{t,f\}for each topic\. Unlike the free\-form LLM descriptor baseline, MonoTM does not generate each topic’s descriptor list freely from top documents\. Instead, the descriptor list is selected from a fixed validated feature vocabulary grounded in document\-embedding activations\.

##### Topic naming for compact qualitative comparison\.

For compact comparison in Tables[12](https://arxiv.org/html/2609.09575#A15.T12),[13](https://arxiv.org/html/2609.09575#A15.T13), and[14](https://arxiv.org/html/2609.09575#A15.T14), we generate high\-level topic names from each descriptor list\. The naming model receives only the descriptor list and the descriptor method; it does not receive the gold label or the candidate\-label list\. For word\-based methods, the naming prompt receives the 30 ranked terms\. For the free\-form LLM descriptor baseline and MonoTM, it receives the 10 semantic descriptors\.

We use the same OpenAI model,gpt\-5\-2025\-08\-07, through the Responses API withreasoning\_effort=low,verbosity=low, JSON\-object output mode, and a maximum output budget of 300 tokens\. We again leave temperature unspecified and retry up to three times if the JSON response cannot be parsed\.

Topic naming promptI have a ranked list of<top words/phrases or semantic descriptors\>derived from a single latent topic\.
Descriptor method:<method\>
Ranked descriptors:<descriptor list\>
Task: Identify the high\-level topic that these descriptors likely came from\.
Rules: Output a concise topic name in 1 to 3 words\. Do not include quotes, explanations, bullets, or extra text\. Return JSON only with this exact schema:\{"name": "\.\.\."\}\.

## Appendix NHigh\-Level Topic Labels Across SAE Configurations

To provide an intuitive qualitative check of MonoTM’s output, we take each topic’s top\-10 interpretable feature labels \(ranked byβt,f\\beta\_\{t,f\}\) and askgpt\-4\.1\-2025\-04\-14\(temperature00\) to produce a 1–3 word topic name using the prompt in Appendix[M](https://arxiv.org/html/2609.09575#A13)\. Table[11](https://arxiv.org/html/2609.09575#A14.T11)shows these names for three SAE configurations \(\(N=1,K=8\)\(N\{=\}1,K\{=\}8\),\(N=3,K=16\)\(N\{=\}3,K\{=\}16\),\(N=5,K=32\)\(N\{=\}5,K\{=\}32\)\)\. For an informative comparison, we align inferred topics to gold categories using the process described in Section[5\.2](https://arxiv.org/html/2609.09575#S5.SS2)\. In many cases, the generated topic labels closely paraphrase the matched gold labels\.

Gold LabelN=1, K=8N=3, K=16N=5, K=3220 Newsgroupsrec\.sport\.hockeyOnline Hockey DiscussionsNHL Hockey FansProfessional Hockey Discoursesoc\.religion\.christianChristian Theology DebatesReligious ControversiesReligion and Moralityrec\.motorcyclesMotorcycle RidingMotorcycle SafetyMotorcycle Culturerec\.sport\.baseballBaseball Online CommunitiesBaseball DiscussionBaseball Analysissci\.cryptCypherpunk MovementCryptography and SurveillanceClipper Chip ControversyWeb of ScienceMedicalHuman Disease ResearchHuman Health ResearchHuman Health DisordersPsychologyChild and Adolescent PsychologyPsychological ResearchSocial PsychologyCSComputer ScienceComputer ScienceComputer SciencebiochemistryBiomedical ScienceBiomedical ResearchBiomedical researchECEControl EngineeringElectrical EngineeringElectrical EngineeringReutersearnQuarterly Financial ReportingFinancial ReportingFinancial Performance ComparisonacqMergers and AcquisitionsFinancial News EventsMergers and Acquisitionsmoney\-fxCentral Bank OperationsBank of England InterventionsCentral Bank InterventionsgrainAgricultural Trade MarketsAgricultural Trade ReportingAgricultural TradecrudeGlobal Oil MarketsGlobal Oil MarketsOPEC Oil MarketsTable 11:Topic labels generated from MonoTM outputs across three SAE configurations\. The gold labels are shown only for orientation and are not provided during topic naming\.
## Appendix OAdditional Descriptor Comparison Results

Tables[12](https://arxiv.org/html/2609.09575#A15.T12)–[14](https://arxiv.org/html/2609.09575#A15.T14)report additional topic\-name tables for the controlled descriptor comparison in Section[5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3)\. As in the main text, all methods describe the same inferred topics under fixed document–topic mixturesΘ\\Theta, and reference labels are shown only for orientation\.

ReferenceMonoTMθ\\theta\-TF\-IDFc\-TF\-IDFLLM\-generatedalt\.atheismReligious debateObjective moralityObjective moralityMeta\-ethicscomp\.graphicsComputer GraphicsImage format conversionImage file formatsComputer Graphicscomp\.os\.ms\-windows\.miscPC hardware troubleshootingHard drive setupPC hard drivesDOS disk configurationcomp\.sys\.ibm\.pc\.hardwareTechnical troubleshootingLead\-acid batteriesLead\-acid batteriesLead\-acid battery storagecomp\.sys\.mac\.hardwarePC hardware troubleshootingMacintosh hardwareMacintosh hardwareMacintosh video hardwarecomp\.windows\.xX11 troubleshootingX11 programmingX11 MotifX11 programmingmisc\.forsaleTech classifiedsComics for saleMarvel ComicsClassifiedsrec\.autosAutomotive discussionCarsCarsPerformance carsrec\.motorcyclesMotorcycle CultureMotorcyclesMotorcyclesMotorcycle riding techniquesrec\.sport\.baseballBaseball AnalyticsBaseballBaseballBaseball sabermetricsrec\.sport\.hockeyIce hockeyNHL PlayoffsNHL hockeyNHL Playoffssci\.cryptClipper Chip DebateClipper ChipClipper ChipClipper Chipsci\.electronicsComputer hardwareElectronic circuitsElectronic circuitsClickless audio switchingsci\.medMedical skepticismCandida yeast syndromeCandida overgrowthCandida overgrowthsci\.spaceSpace commercializationLunar explorationSpaceflightLunar habitation prizesoc\.religion\.christianTheological debatesChristianityChristian theologyBiblical hermeneuticstalk\.politics\.gunsWaco siegeWaco siegeWaco siegeWaco Siegetalk\.politics\.mideastEthno\-national conflictsArmenian GenocideArmenian GenocideArmenian genocide controversytalk\.politics\.miscAmerican culture warsGun rightsGun rightsGay rights debatetalk\.religion\.miscUsenet discussionsEmailUsenet newsgroupsMailing list administrationTable 12:High\-level topic names generated from descriptor outputs on 20 Newsgroups\. All methods describe the same inferred topics under fixed document–topic mixturesΘ\\Theta\. Reference labels are shown only for orientation and are not provided to the topic\-naming model\.ReferenceMonoTMθ\\theta\-TF\-IDFc\-TF\-IDFLLM\-generatedbiochemistryBiomedical researchCancer cell signalingMolecular oncologySignal transductionCivilCivil and Environmental EngineeringRainwater harvestingRainwater HarvestingRainwater harvestingCSComputer Science ResearchBig Data ComputingBig DataCloud and Distributed SystemsECEControl Systems EngineeringPower converter controlPower Converter ControlDigital Inverter ControlMAEComputational mechanicsComposite MaterialsComposite materialsMechanical behavior of materialsMedicalClinical epidemiologyMale hypogonadismMale hypogonadismMale hypogonadismPsychologyPsychology researchParenting and child behaviorChild DevelopmentProsocial BehaviorTable 13:High\-level topic names generated from descriptor outputs on Web of Science\. All methods describe the same inferred topics under fixed document–topic mixturesΘ\\Theta\. Reference labels are shown only for orientation and are not provided to the topic\-naming model\.ReferenceMonoTMθ\\theta\-TF\-IDFc\-TF\-IDFLLM\-generatedearnEarnings reportsEarnings ReportsEarnings reportsCorporate EarningsacqMergers and AcquisitionsBank mergersBank MergersBank Mergersmoney\-fxMoney market interventionsMoney market operationsMoney marketOpen market operationsgrainAgricultural TradeExport Enhancement ProgramGrain export subsidiesExport Enhancement ProgramcrudeOPEC oil marketOil productionOPEC oil productionEcuador oil crisistradeUS\-Japan Trade DisputesUS\-Japan trade disputeUS\-Japan trade disputeU\.S\.\-Japan trade disputeinterestInterest Rate ChangesPrime rateInterest ratesPrime rate changesshipPersian Gulf conflictPersian Gulf Tanker WarGulf Tanker WarOperation Nimble ArcherwheatCorporate financial distressEarnings reportsEarnings reportsNonrecurring itemscornMergers and AcquisitionsRetail mergers and acquisitionsBusiness newsConsumer retail M&AoilseedFinancial reportingCorporate earningsCorporate earningsCorporate earningssugarAgricultural tradeSugar crop productionCrop productionGlobal sugar productiondlrInternational monetary policyLouvre AccordLouvre AccordG7 currency coordinationgnpMacroeconomic IndicatorsEconomic GrowthMacroeconomic IndicatorsOECD economic outlookcoffeeInternational Coffee TradeInternational coffee tradeInternational Coffee AgreementInternational Cocoa Agreementveg\-oilTrade and M&AMergers and acquisitionsMergers and acquisitionsHBJ takeover battlemoney\-supplyFederal Reserve operationsOpen market operationsOpen market operationsOpen market operationsgoldMining Industry FinanceGold miningGold miningGold miningnat\-gasAirline MergersAirline mergersAirline mergersTWA USAir takeoverlivestockUS Agricultural PolicyLivestock futuresLivestock futuresMeatpacking labor disputesTable 14:High\-level topic names generated from descriptor outputs on the 20 most frequent Reuters labels\. Because Reuters is multi\-label and imbalanced, reference labels should be read as orientation labels rather than exhaustive topic descriptions\.
## Appendix PRepresentative Descriptor Lists

Table[15](https://arxiv.org/html/2609.09575#A16.T15)provides representative descriptor lists before topic\-name compression for the controlled descriptor comparison in Section[5\.3\.3](https://arxiv.org/html/2609.09575#S5.SS3.SSS3)\. All methods describe the same fixed document–topic mixturesΘ\\Theta\. The examples illustrate the different roles of the descriptor mechanisms: word\-based methods expose lexical anchors, free\-form LLM\-generated descriptors often summarize salient document clusters, and MonoTM descriptors are selected from a fixed vocabulary of validated SAE features ranked by a model\-estimated topic–feature distribution\.

ReferenceMethodTop descriptorssci\.cryptMonoTMDistrust of government and defense of liberty; government skepticism, privacy concerns, argumentative discourse; technical discourse on cryptographic algorithms and implementation; secure voice tech, compression, early internet security; government surveillance and cryptographic privacy concernssci\.cryptθ\\theta\-TF\-IDFclipper; encryption; keys; escrow; chip; government; technology; secure; nsa; keysci\.cryptc\-TF\-IDFencryption; clipper; chip; government; technology; key; keys; escrow; nsa; securesci\.cryptLLM\-generatedGovernment key\-escrowed Clipper Chip proposal; secret Skipjack algorithm and family key; law enforcement access via escrowed unit keys; LEAF transmits session key encrypted for authorities; critics warn of mandated, tappable encryption standardtalk\.politics\.miscMonoTMDistrust of government and defense of liberty; contentious debate surrounding homosexuality and morality; moral debates about sexuality and statistics; gun control, self\-defense arguments, statistical debate; contentious social debate, group identity, and conflicttalk\.politics\.miscθ\\theta\-TF\-IDFcramer; optilink; clayton; homosexuals; people; gay; gun; militia; rights; sexualtalk\.politics\.miscc\-TF\-IDFpeople; cramer; edu; optilink; militia; homosexuals; gay; gun; sexual; rightstalk\.politics\.miscLLM\-generatedDebating Kinsey versus Guttmacher homosexuality prevalence statistics; claims homosexuals exaggerate numbers to sway politicians; arguments over gay rights anti\-discrimination employment laws; libertarian stance: private hiring without government interference; accusations linking gays with child molestation, NAMBLAECEMonoTMEngineered control systems and PID control design; electrical power system analysis and control; high\-performance electric motor design and analysis; analog circuit design and performance metrics; engineered system optimization and energy efficiency analysisECEθ\\theta\-TF\-IDFconverter; control; digital; voltage; proposed; power; current; controller; circuit; systemECEc\-TF\-IDFcontrol; proposed; voltage; converter; power; current; controller; circuit; digital; systemECELLM\-generatedFPGA\-based digital control for power converters; digital PLLs for single\-phase grid synchronization; discrete\-time modulation introducing transport delays; finite control set MPC implemented fully on FPGA; deadbeat predictive current control in invertersCivilMonoTMApplied hydrology, water resource engineering, rainwater harvesting; engineered system optimization and energy efficiency analysis; sustainable building and environmental responsibility concepts; construction project management, analytical methodology, performance evaluation; water pollution assessment and scientific reportingCivilθ\\theta\-TF\-IDFwater; rainwater; rwh; harvesting; demand; tank; system; saving; efficiency; supplyCivilc\-TF\-IDFwater; rainwater; rwh; harvesting; use; demand; tank; saving; efficiency; systemCivilLLM\-generatedRainwater harvesting supplying toilets, laundry, irrigation; economic feasibility via payback and NPV analyses; optimal storage tank sizing using daily simulations; water saving efficiency and reliability metrics; integrating rainwater with greywater reuse systemscrudeMonoTMOil price stability and OPEC market control; OPEC oil production, pricing, and market regulation; OPEC oil price and production control; OPEC oil production, pricing, and regulation reports; oil market data and energy economicscrudeθ\\theta\-TF\-IDFbpd; oil; ecuador; opec; barrels; crude; prices; pipeline; production; exportscrudec\-TF\-IDFsaid; oil; bpd; opec; ecuador; crude; barrels; pipeline; production; pricescrudeLLM\-generatedEarthquake halts Ecuador crude exports, declares force majeure; Lago Agrio–Balao pipeline severely damaged, costly repairs; temporary pipeline link to Colombia’s Tumaco port; Venezuela lends Ecuador crude for exports, domestic needs; Ecuador plans over\-quota OPEC production to repay loansshipMonoTMGovernment economic intervention and trade policy; US\-Iran military conflict and retaliation narratives; international military conflict and geopolitical tension; Persian Gulf naval conflict and escalation threat; Persian Gulf oil security, U\.S\. military responseshipθ\\theta\-TF\-IDFiran; gulf; iranian; attack; kuwaiti; oil; reflagged; ship; tanker; platformshipc\-TF\-IDFiran; said; gulf; iranian; oil; attack; kuwaiti; ship; platform; tankershipLLM\-generatedU\.S\. destroyers shell Iranian Rostam oil platform; retaliation for Silkworm strike on Sea Isle City; Navy raids second platform, destroys radar communications; Iran vows crushing retaliation for platform attacks; escorting reflagged Kuwaiti tankers through GulfTable 15:Longer representative descriptor lists before topic\-naming\. Word\-based descriptors often expose terms or named entities; LLM\-generated descriptors often describe salient document clusters; MonoTM descriptors are selected from validated SAE features and ranked by a topic–feature distribution\.
## Appendix QAffordance Demonstration: Topic Relations in a Shared Semantic\-Feature Space

This appendix demonstrates a downstream affordance of MonoTM’s shared semantic\-feature representation\. The goal is not to introduce a new quantitative benchmark, but to illustrate how MonoTM can support analyses beyond compact topic description\. Because MonoTM represents both documents and topics in a shared vocabulary of validated SAE features, users can inspect which features are shared across documents, which features distinguish neighboring topics, and which semantic attributes cut across topic boundaries\. We demonstrate this affordance by analyzing how topics are related in the Web of Science corpus\.

##### Construction details\.

We perform the analysis on Web of Science under the same fixed document–topic mixturesΘ\\Thetaused in the controlled descriptor comparison\. For MonoTM, we use the estimated topic–feature distributionBBand retain the top 200 validated features per topic\. We keep the rawβt,f\\beta\_\{t,f\}masses rather than renormalizing them\. For the lexical analogue, we use the top 50 terms per topic from theθ\\theta\-weighted TF\-IDF topic–term representation, with term scores normalized to sum to one within each topic\.

For two topicsiiandjjwith MonoTM topic–feature distributionsβi\\beta\_\{i\}andβj\\beta\_\{j\}, we define their shared feature mass as

O⁡\(i,j\)=∑fmin⁡\(βi,f,βj,f\)\.O\(i,j\)=\\sum\_\{f\}\\min\(\\beta\_\{i,f\},\\beta\_\{j,f\}\)\.More generally, for either representation, we compute topic\-pair overlap as

O⁡\(i,j\)=∑xmin⁡\(vi,x,vj,x\),O\(i,j\)=\\sum\_\{x\}\\min\(v\_\{i,x\},v\_\{j,x\}\),wherexxindexes validated SAE features for MonoTM and lexical terms for the TF\-IDF analogue\. To explain each edge, we rank shared items bymin⁡\(vi,x,vj,x\)\\min\(v\_\{i,x\},v\_\{j,x\}\)and report the top shared features or terms\. The visualized networks display the eight strongest topic\-pair overlaps, and the tables below report representative shared items from those edges\.

To visualize the resulting topic relations, Figure[9](https://arxiv.org/html/2609.09575#A17.F9)shows the strongest topic\-pair overlaps as networks for MonoTM and theθ\\theta\-weighted TF\-IDF analogue\. Figure[10](https://arxiv.org/html/2609.09575#A17.F10)provides the corresponding full pairwise overlap matrices, showing whether the strongest network edges reflect isolated pairwise relations or broader overlap structure across topics\.

Figure 9:Web of Science topic relation networks under fixed document–topic mixtures\. Edges show the strongest topic\-pair overlaps, with edge width proportional to shared mass\. \(a\) MonoTM computes overlap in the validated SAE\-feature space, so edge labels identify the semantic features that explain each topic connection\. \(b\) Aθ\\theta\-weighted TF\-IDF analogue computes overlap in a lexical term space, so edge labels identify shared keywords\. The comparison illustrates how MonoTM supports feature\-level interpretation of topic relations beyond lexical overlap\.![Refer to caption](https://arxiv.org/html/2609.09575v1/MonoTM_topic_relation_overlap_heatmap_revised.png)

![Refer to caption](https://arxiv.org/html/2609.09575v1/tfidf_topic_relation_overlap_heatmap_revised.png)

Figure 10:Topic\-pair overlap matrices for Web of Science\. Left: MonoTM shared\-feature mass using the top 200 validated features per topic fromBB\. Right:θ\\theta\-weighted TF\-IDF shared\-term mass using the top 50 lexical terms per topic\. Diagonal entries are set to zero\.
##### Qualitative comparison\.

Figures[9](https://arxiv.org/html/2609.09575#A17.F9)and[10](https://arxiv.org/html/2609.09575#A17.F10)show that the two representations yield qualitatively different topic\-relation structures and edge explanations\. Table[16](https://arxiv.org/html/2609.09575#A17.T16)reports representative shared items for several high\-overlap topic pairs\. MonoTM links topics through semantic features such as clinical disease investigation, severe mental illness and treatment, efficient algorithm implementation and hardware optimization, applied engineering education, and engineered system optimization and energy efficiency\. By contrast, theθ\\theta\-weighted TF\-IDF analogue often explains topic links using lexical items such asmodel,systems,method,study,results,paper, andbased\. Thus, a word\-based representation can indicate that two topics overlap lexically, but MonoTM can make the semantic basis of the relation more explicit\.

Topic pairMonoTM shared validated featuresθ\\theta\-TF\-IDF shared termsbiochemistry–MedicalPulmonary disease research and clinical investigation focus\.; Cancer research, molecular mechanisms, and clinical outcomes\.; Gastrointestinal distress, IBS/IBD, patient well\-beingstudyMedical–PsychologySevere mental illness, clinical diagnosis, and treatment\.; Anxiety, depression, and psychological distress research\.; Child development, family well\-being, healthcare support\.study; results; levelsCS–ECEEfficient algorithm implementation and hardware optimization\.; Applied technology for infrastructure and efficiency\.; Mathematical modeling and algorithmic development in data sciencepaper; based; model; performance; proposedMAE–ECEMagnetic material design and analysis in engineering\.; Precise sensor measurement and instrumentation details\.; Detailed thermodynamic system analysis and optimization\.model; experimental; method; using; resultsCivil–ECEEngineered system optimization and energy efficiency analysis\.; Engineered thermal systems and heat transfer analysis\.; Applied technology for infrastructure and efficiency\.model; systems; performance; usingMAE–CivilEngineered thermal systems and heat transfer analysis\.; Applied engineering education and curriculum design\.; Technical design, optimization, and performance analysis\.model; usingTable 16:Representative shared items explaining Web of Science topic\-pair edges\. MonoTM edge explanations are validated SAE features ranked by sharedβ\\betamass, while the lexical analogue explains edges through sharedθ\\theta\-weighted TF\-IDF terms\.Overall, this demonstration illustrates a broader affordance of MonoTM: the same validated feature vocabulary can support topic description, topic comparison, and inspection of semantic attributes that cut across benchmark category boundaries\.