Modeling Document and Section Hierarchies in Embedding Space
Summary
Columbia University researchers introduce the Hierarchical Embedded Topic Model (hETM), which couples document- and section-level topic distributions via a hierarchical prior in embedding space, with amortized variational inference showing improved perplexity, topic quality, and clustering across legal, financial, and forum corpora.
View Cached Full Text
Cached at: 11/19/26, 03:03 AM
# Modeling Document and Section Hierarchies in Embedding Space Source: [https://arxiv.org/html/2611.21980](https://arxiv.org/html/2611.21980) Sharyn O’HalloranKshitij ChandnaAffiliation:Columbia UniversityEmail:[mailto:](mailto:)[kriste\.krstovski@columbia\.edu](mailto:[email protected]) ###### Abstract Embedded topic models integrate word embeddings into the topic generation process, producing coherent topics and robust performance on large vocabularies\. However, each document is represented by a single topic distribution, ignoring its internal structure\. Many real world collections, including legal documents, company and government reports, and online discussion forums, exhibit hierarchical organization in which documents consist of multiple sections that often address distinct subtopics\. We introduce the Hierarchical Embedded Topic Model \(hETM\), a structured topic model that couples the topic distribution of a document and those of its sections through a hierarchical prior in the embedding space\. We derive an amortized variational inference procedure for hETM and evaluate it on structured collections from each of these domains\. Experimental results show that hETM achieves the lowest or near\-lowest held\-out perplexity among flat and hierarchical topic model baselines, along with comparable or higher topic quality\. It also performs strongly on a downstream clustering task, achieving the best clustering performance on most of the collections\. ## 1Introduction Bag of words \(BOW\) models discard grammar and rely only on word co\-occurrence statistics\. Topic models, from latent Dirichlet allocation \(LDA\)\([Blei et al\., 2003](https://arxiv.org/html/2611.21980#bib.bib1)\)to recent neural variants\([Dieng et al\., 2020](https://arxiv.org/html/2611.21980#bib.bib3)\), all adopt this representation\. The way a corpus is split into documents therefore matters: long documents, such as books or statutes, yield diffuse statistics, while short ones, such as forum posts, yield sparsity\. Yet many real world collections are explicitly hierarchical: a document is a structured sequence of sections, and meaning lives at both levels\. This structure is clearest in the*legal*domain\. Statutes and bills are divided into sections, and contracts into clauses: a single provision covers one topic, but its meaning depends on the bill it belongs to, since large bills often mix trade or tax provisions with unrelated ones\. In*finance*, company disclosures are organized into predefined sections, such as the “Risk Factors” or “Management’s Discussion and Analysis” sections of a 10\-K and the segments of an earnings call; analysts read these one at a time, for example to track a firm’s liquidity risk, but still need a summary of the whole filing\. In*online forums*, a discussion thread is a sequence of short posts grouped into boards or newsgroups, as in the 20 Newsgroups collection\([Lang, 1995](https://arxiv.org/html/2611.21980#bib.bib26)\), and a single post says little without the thread it belongs to\. In each case, the section and the document are two levels of the same object, and a single topic distribution per document captures only one of them\. Treating such documents as flat bags of words forces a choice of unit at a cost: modeling the whole document blurs the distinct topics of its sections, while modeling each section on its own loses statistical strength\. A hierarchy that ties each section’s topics to its document recovers both: the section topics stay sharp, yet still combine into a document level theme\. This matters most for long, varied documents with large, specialized vocabularies of legal citations, financial and scientific terms, and rare named entities — exactly where flat LDA breaks down and embedding based topics help — which calls for a model that captures both the document hierarchy and the vocabulary’s embedding structure\. Two largely separate lines of work address these limitations\. The first incorporates word embeddings: the embedded topic model \(ETM\) of[Dieng et al\. \(2020\)](https://arxiv.org/html/2611.21980#bib.bib3)represents each topic as a point in a word embedding space and yields coherent topics on heavy\-tailed vocabularies, and recent neural topic models build hierarchies of topics within that space\([Duan et al\., 2021](https://arxiv.org/html/2611.21980#bib.bib19);[Xu et al\., 2022](https://arxiv.org/html/2611.21980#bib.bib20);[Wu et al\., 2024b](https://arxiv.org/html/2611.21980#bib.bib21)\)\. These models assume a single topic distribution per document\. The second improves the structure of the prior:[Wallach et al\. \(2009\)](https://arxiv.org/html/2611.21980#bib.bib11)use asymmetric Dirichlet priors, and[Krstovski et al\. \(2016\)](https://arxiv.org/html/2611.21980#bib.bib7)introduce mlhLDA, which couples per\-section topic proportions through a document\-level hierarchical Dirichlet prior but inherits LDA’s bag of words likelihood and its degradation on large vocabularies\. hETM focuses on the second kind of hierarchy\. We model how a document’s topic proportions relate to those of its sections, rather than imposing a parent–child structure among the topics themselves\. The topic\-tree hierarchies cited above are complementary to this coupling and could be layered on top\. Combining embeddings with document\-structure coupling is not straightforward\. ETM places a logistic\-normal prior on topic proportions and can be fit using the reparameterization trick[Kingma and Welling \(2014\)](https://arxiv.org/html/2611.21980#bib.bib4), while mlhLDA uses a nested Dirichlet prior that lets the document proportion serve as the base measure for the section\. Swapping embedded topics under that prior forfeits ETM’s amortized inference\. We propose the Hierarchical Embedded Topic Model \(hETM\): rather than importing mlhLDA’s Dirichlet hierarchy, we import its coupling, drawing each section proportion as a Gaussian perturbation of its document proportion in logit space\. This preserves the shrinkage toward the document while keeping the model reparameterizable\. Our contribution is twofold: \(1\) hETM, which couples section and document topic proportions through a hierarchical logistic\-normal prior; and \(2\) an amortized variational inference procedure with a closed\-form level\-coupling term, in which the Gaussian precision plays the role of the Dirichlet concentration in mlhLDA\. We evaluate hETM on four collections across three domains: U\.S\. trade legislation and European Union \(EU\) legislation \(legal\), corporate annual reports \(finance\), and the 20 Newsgroups corpus \(online forums\)\. hETM attains lower perplexity than flat and hierarchical topic models, comparable or higher topic quality, and the highest clustering accuracy on most datasets\. We review prior work, present hETM and its inference, and evaluate it intrinsically and extrinsically\. ## 2Background and Related Work Hierarchical prior structures for topic models have received attention in the past\. For example,[Wallach et al\. \(2009\)](https://arxiv.org/html/2611.21980#bib.bib11)showed that an asymmetric Dirichlet prior, placed as a hyperprior over the document\-topic distributions, provides better modeling of documents across the whole collection than the symmetric priors used in the original LDA\. Other approaches build a tree over the topics themselves\. The nested Chinese restaurant process\([Blei et al\., 2010](https://arxiv.org/html/2611.21980#bib.bib28)\)assigns each document a path down an unbounded topic tree, Pachinko allocation\([Li and McCallum, 2006](https://arxiv.org/html/2611.21980#bib.bib27)\)captures topic correlations through a directed acyclic graph of topics, and more recently[Kim et al\. \(2013\)](https://arxiv.org/html/2611.21980#bib.bib6)introduced tiLDA, a topic model of document collections with nested hierarchies\. While these approaches model richer structure than LDA, they rely on Gibbs sampling or batch variational Bayes \(VB\) inference and, as such, require multiple iterations over the whole collection\. More closely related to our work,[Krstovski et al\. \(2016\)](https://arxiv.org/html/2611.21980#bib.bib7)introduced mlhLDA, a topic model for documents with a predetermined section structure\. Rather than a single document\-topic distribution, mlhLDA assigns each section its own topic distribution and couples these to a document level distribution through a hierarchical Dirichlet prior, so that the topics of a section are shrunk toward the topics of the document that contains it\. mlhLDA is trained with online VB\([Hoffman et al\., 2010](https://arxiv.org/html/2611.21980#bib.bib5)\), requires only a single pass over the collection, and extends to multilingual collections \(mlhPLTM\) by sharing a tuple\-level distribution across languages\. As with the approaches above, however, mlhLDA represents each topic as a separate distribution over the vocabulary and, like LDA, does not perform well on large vocabularies\. A separate line of work replaces the Dirichlet\-multinomial of LDA with neural networks \(NNs\)\. For example,[Miao et al\. \(2016\)](https://arxiv.org/html/2611.21980#bib.bib9)and[Srivastava and Sutton \(2017\)](https://arxiv.org/html/2611.21980#bib.bib10)cast topic modeling as amortized variational inference, in which an inference network encodes a document’s BOW and a decoder reconstructs it\. More recently,[Dieng et al\. \(2020\)](https://arxiv.org/html/2611.21980#bib.bib3)introduced the embedded topic model \(ETM\), which represents each topic as a point in a word embedding space and draws words using the inner product between the topic embedding and the word embeddings\. Because it relies on word embeddings, the ETM produces coherent topics even on large vocabularies, a regime where LDA degrades, and because it places a logistic\-normal rather than a Dirichlet prior on the topic proportions, it can be trained with the reparameterization trick\. A related embedding based model is Gaussian\-LDA\([Das et al\., 2015](https://arxiv.org/html/2611.21980#bib.bib2)\), which generates word embeddings rather than words; an earlier embedding\-based topic model, TopicVec\([Li et al\., 2016](https://arxiv.org/html/2611.21980#bib.bib36)\), also represents topics as points in the word embedding space and combines them with LDA\-style Dirichlet topic proportions via variational expectation\-maximization\. Both models keep a single topic distribution per document, as does ETM\. Several recent neural topic models add hierarchy to this embedding based view;[Wu et al\. \(2024a\)](https://arxiv.org/html/2611.21980#bib.bib23)give a recent survey of the area\.[Isonuma et al\. \(2020\)](https://arxiv.org/html/2611.21980#bib.bib22)introduce a tree structured neural topic model that parameterizes an unbounded topic tree with doubly recurrent networks\.[Duan et al\. \(2021\)](https://arxiv.org/html/2611.21980#bib.bib19)propose SawETM, which stacks topic embedding layers in a deep gamma belief network, and[Xu et al\. \(2022\)](https://arxiv.org/html/2611.21980#bib.bib20)introduce HyperMiner, which places word and topic embeddings in hyperbolic space so that its tree\-like geometry captures the hierarchy among topics\.[Wu et al\. \(2024b\)](https://arxiv.org/html/2611.21980#bib.bib21)propose TraCo, which learns a topic tree through a transport\-plan dependency between parent and child topic distributions and separates topic hierarchy from topic diversity\. A separate, non\-generative line embeds document sections and clusters them to read off a hierarchy after the fact\([Viegas et al\., 2025](https://arxiv.org/html/2611.21980#bib.bib24)\); such approaches, however, do not define a likelihood or a principled coupling between levels\. Sub\-document topic modeling, which assigns a topic distribution to each sub\-document unit rather than to the document as a whole, is a separate line with its own history\([Wang et al\., 2009](https://arxiv.org/html/2611.21980#bib.bib31);[Balikas et al\., 2016](https://arxiv.org/html/2611.21980#bib.bib32);[Li et al\., 2017](https://arxiv.org/html/2611.21980#bib.bib33);[Jiang et al\., 2019](https://arxiv.org/html/2611.21980#bib.bib34)\), with SubNTM\([Nguyen et al\., 2025](https://arxiv.org/html/2611.21980#bib.bib35)\)as the most recent and the closest to our setting\. SubNTM places the between\-level coupling in the generative equation, through a document\-independent noise vector, whereas hETM places it in the prior itself, drawing each section’s topic distribution around that of the document that contains it\. Unlike the neural models above, which build a hierarchy over the topics themselves, our model builds a hierarchy over the document sections\. A topic hierarchy of the kind learned by[Xu et al\. \(2022\)](https://arxiv.org/html/2611.21980#bib.bib20)or[Wu et al\. \(2024b\)](https://arxiv.org/html/2611.21980#bib.bib21)is complementary and could in principle be layered on top\. To our knowledge, ours is the first generative model in which the section and document levels share a single embedded topic space and are coupled in the prior\. ## 3Modeling Topic Hierarchies in Embedding Space hETM applies to documents that are split into sections, giving a parent–child relationship between a document and its parts\. The split can be explicit, as in legislative bills, EU regulations, the items of company reports, or implicit, as when several short documents are grouped into a larger unit, such as forum posts on a common theme\. We refer to these parts as sections throughout\. A document and its sections describe topics at two levels\. The document as a whole has an overall theme, while each section focuses on a narrower set of topics: a trade bill is broadly about commerce, but one section defines terms, another sets tariff schedules, and a third lists penalties\. A model that captures only the document level blurs these sections together, while one that treats each section on its own loses the shared subject that ties them\. hETM models both levels jointly\. Each section has its own distribution over topics, and these distributions are drawn around a single document distribution\. Because the two levels are coupled rather than estimated separately, the section topics stay sharp while the document distribution emerges as a coherent summary of them\. We consider a collection ofDDdocuments, where each documentdi=\(si,1,…,si,ni\)d\_\{i\}=\(s\_\{i,1\},\\dots,s\_\{i,n\_\{i\}\}\)is an ordered list ofnin\_\{i\}sections, and each section is a BOW over a vocabulary of sizeVV\. The ordering indexes the sections but is not used by the model, which treats sections as conditionally independent given the document representation; incorporating section order would require a sequential prior and is beyond our scope\. We denotexi,jx\_\{i,j\}as the word counts of sectionsi,js\_\{i,j\}andxi=∑jxi,jx\_\{i\}=\\sum\_\{j\}x\_\{i,j\}as the document counts, and model the collection withKKtopics\. As in embedding based topic models, hETM represents each of theKKtopics not as a separate distribution over the vocabulary but as a pointαk∈ℝL\\alpha\_\{k\}\\in\\mathbb\{R\}^\{L\}in a shared word embedding space, whereρ∈ℝL×V\\rho\\in\\mathbb\{R\}^\{L\\times V\}is the matrix of word embeddings\. The topic\-word distributionβk\\beta\_\{k\}is the softmax of the inner products between the topic and word embeddings, βk=softmax\(ρ⊤αk\),\\beta\_\{k\}=\\softmax\(\\rho^\{\\top\}\\alpha\_\{k\}\),\(1\) which lets hETM produce coherent topics on large vocabularies\. ### 3\.1Generative Process hETM generates a document together with its sections\. Each document is summarized by a document\-topic distribution, and each of its sections by its own section\-topic distribution over the sameKKtopics\. Rather than drawing these independently, we draw the section distributions as perturbations of the document distribution, tying the two levels together\. To keep the model reparameterizable, we place this hierarchy in the logit space: the logit of a section is drawn from a Gaussian centered on the logit of its document, and the corresponding proportions are the softmax of that logit\. For each documentdid\_\{i\}, hETM assumes the following generative process: - •Choose a document logitδid∼𝒩\(0,σd2I\)\\delta\_\{i\}^\{d\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{d\}^\{2\}I\)and setθid=softmax\(δid\)\\theta\_\{i\}^\{d\}=\\softmax\(\\delta\_\{i\}^\{d\}\)\. - •For each sectionsi,js\_\{i,j\}in documentdid\_\{i\}: - –Choose a section logitδi,js∼𝒩\(δid,σs2I\)\\delta\_\{i,j\}^\{s\}\\sim\\mathcal\{N\}\(\\delta\_\{i\}^\{d\},\\sigma\_\{s\}^\{2\}I\)and setθi,js=softmax\(δi,js\)\\theta\_\{i,j\}^\{s\}=\\softmax\(\\delta\_\{i,j\}^\{s\}\)\. - –For each wordwwin sectionsi,js\_\{i,j\}: - \*Choose a topicz∼Cat\(θi,js\)z\\sim\\mathrm\{Cat\}\(\\theta\_\{i,j\}^\{s\}\)\. - \*Choose a wordw∼Cat\(βz\)w\\sim\\mathrm\{Cat\}\(\\beta\_\{z\}\)\. The document logitδid\\delta\_\{i\}^\{d\}serves as the mean of the section logitsδi,js\\delta\_\{i,j\}^\{s\}, so each section’s topics vary around the document’s\. The section varianceσs2\\sigma\_\{s\}^\{2\}controls the coupling: small values shrink sections toward the document, large values let them vary freely\. As in embedding based topic models, the topic assignmentzzcan be marginalized out, so that the probability of a word depends only on the section logit, p\(wi,j,n∣δi,js\)=\(θi,js\)⊤β⋅,wi,j,n,p\(w\_\{i,j,n\}\\mid\\delta\_\{i,j\}^\{s\}\)=\(\\theta\_\{i,j\}^\{s\}\)^\{\\top\}\\beta\_\{\\cdot,w\_\{i,j,n\}\},\(2\)and the joint probability of a document and its logits is p\(di,δdi,δsi,1:ni\)=𝒩\(δid;0,σd2I\)×∏j𝒩\(δi,js,δid,σs2I\)∏n\(θi,js\)⊤β⋅,wi,j,n\.\\begin\{split\}p\(d\_\{i\},&\\,\\delta^\{d\}\_\{i\},\\delta^\{s\}\_\{i,1:n\_\{i\}\}\)=\\mathcal\{N\}\(\\delta\_\{i\}^\{d\};0,\\sigma\_\{d\}^\{2\}I\)\\\\ &\\times\\prod\_\{j\}\\mathcal\{N\}\(\\delta\_\{i,j\}^\{s\};\\delta\_\{i\}^\{d\},\\sigma\_\{s\}^\{2\}I\)\\prod\_\{n\}\(\\theta\_\{i,j\}^\{s\}\)^\{\\top\}\\beta\_\{\\cdot,w\_\{i,j,n\}\}\.\\end\{split\}\(3\) ## 4Amortized Variational Inference The posterior over the document and section logits is intractable, so we approximate it with amortized variational inference\. We share two inference networks across the corpus: a document encoder that maps the normalized document countsx¯i=xi/Ni\\bar\{x\}\_\{i\}=x\_\{i\}/N\_\{i\}to a diagonal Gaussian over the document logit, and a section encoder that maps the normalized section countsx¯i,j\\bar\{x\}\_\{i,j\}to a diagonal Gaussian over each section logit\. The variational posterior mirrors the generative factorization, q\(δid,δsi,1:ni\)=qνd\(δid∣xi\)×∏jqνs\(δi,js∣xi,j\),\\begin\{split\}q\(\\delta\_\{i\}^\{d\},\\delta^\{s\}\_\{i,1:n\_\{i\}\}\)&=q\_\{\\nu\_\{d\}\}\(\\delta\_\{i\}^\{d\}\\mid x\_\{i\}\)\\\\ &\\quad\\times\\prod\_\{j\}q\_\{\\nu\_\{s\}\}\(\\delta\_\{i,j\}^\{s\}\\mid x\_\{i,j\}\),\\end\{split\}\(4\) with\(μid,log\(\(σid\)2\)\)=NNd\(x¯i,νd\)\(\\mu\_\{i\}^\{d\},\\ \\log\\\!\\big\(\(\\sigma\_\{i\}^\{d\}\)^\{2\}\\big\)\)=\\mathrm\{NN\}\_\{d\}\(\\bar\{x\}\_\{i\};\\nu\_\{d\}\)and\(μi,js,log\(σi,js\)2\)=NNs\(x¯i,j,νs\)\(\\mu\_\{i,j\}^\{s\},\\log\(\\sigma\_\{i,j\}^\{s\}\)^\{2\}\)=\\mathrm\{NN\}\_\{s\}\(\\bar\{x\}\_\{i,j\};\\nu\_\{s\}\)\. Because the document encoder reads the aggregate countsx¯i\\bar\{x\}\_\{i\}, the document posterior draws on the combined word statistics of all its sections\. We fit the encoder and model parameters by maximizing an evidence lower bound \(ELBO\)\. For each document, the bound separates into a reconstruction term and two KL terms,ℒi=\(R\)−\(D\)−\(S\)\\mathcal\{L\}\_\{i\}=\\mathrm\{\(R\)\}\-\\mathrm\{\(D\)\}\-\\mathrm\{\(S\)\}, where the reconstruction term and the document KL term are \(R\)\\displaystyle\\mathrm\{\(R\)\}=∑j𝔼q\(δi,js\)\[∑vxi,j,vlog\(θi,js\)⊤β⋅,v\],\\displaystyle=\\sum\_\{j\}\\mathbb\{E\}\_\{q\(\\delta^\{s\}\_\{i,j\}\)\}\\\!\\Big\[\\textstyle\\sum\_\{v\}x\_\{i,j,v\}\\log\(\\theta^\{s\}\_\{i,j\}\)^\{\\top\}\\\!\\beta\_\{\\cdot,v\}\\Big\],\(5\)\(D\)\\displaystyle\\mathrm\{\(D\)\}=12∑k\[\(μi,kd\)2\+\(σi,kd\)2σd2−1−log\(σi,kd\)2σd2\]\.\\displaystyle=\\tfrac\{1\}\{2\}\\\!\\sum\_\{k\}\\\!\\Big\[\\tfrac\{\(\\mu^\{d\}\_\{i,k\}\)^\{2\}\+\(\\sigma^\{d\}\_\{i,k\}\)^\{2\}\}\{\\sigma\_\{d\}^\{2\}\}\-1\-\\log\\tfrac\{\(\\sigma^\{d\}\_\{i,k\}\)^\{2\}\}\{\\sigma\_\{d\}^\{2\}\}\\Big\]\.\(6\)The section term is the only nonstandard one: the prior on a section logit,𝒩\(δid,σs2I\)\\mathcal\{N\}\(\\delta^\{d\}\_\{i\},\\sigma\_\{s\}^\{2\}I\), has a random meanδid\\delta^\{d\}\_\{i\}, so we take its expectation underq\(δid\)q\(\\delta^\{d\}\_\{i\}\)\. Using𝔼\[\(μs−δd\)2\]=\(μs−μd\)2\+\(σd\)2\\mathbb\{E\}\[\(\\mu^\{s\}\-\\delta^\{d\}\)^\{2\}\]=\(\\mu^\{s\}\-\\mu^\{d\}\)^\{2\}\+\(\\sigma^\{d\}\)^\{2\}, the section KL term becomes \(S\)=12∑j∑k\[\(σi,j,ks\)2\+\(μi,j,ks−μi,kd\)2\+\(σi,kd\)2σs2−1\+logσs2−log\(σsi,j,k\)2\]\.\\begin\{split\}\\mathrm\{\(S\)\}=\\tfrac\{1\}\{2\}\\sum\_\{j\}\\sum\_\{k\}\\Big\[&\\tfrac\{\(\\sigma^\{s\}\_\{i,j,k\}\)^\{2\}\+\(\\mu^\{s\}\_\{i,j,k\}\-\\mu^\{d\}\_\{i,k\}\)^\{2\}\+\(\\sigma^\{d\}\_\{i,k\}\)^\{2\}\}\{\\sigma\_\{s\}^\{2\}\}\\\\\[\-1\.0pt\] &\-1\+\\log\\sigma\_\{s\}^\{2\}\-\\log\(\\sigma^\{s\}\_\{i,j,k\}\)^\{2\}\\Big\]\.\\end\{split\}\(7\)The cross term\(μs−μd\)2/σs2\(\\mu^\{s\}\-\\mu^\{d\}\)^\{2\}/\\sigma\_\{s\}^\{2\}pulls each section mean toward its document mean with strength1/σs21/\\sigma\_\{s\}^\{2\}, so a smallerσs2\\sigma\_\{s\}^\{2\}ties sections more tightly to the document; the term\(σd\)2/σs2\(\\sigma^\{d\}\)^\{2\}/\\sigma\_\{s\}^\{2\}accounts for the document’s own posterior uncertainty and makes the expression exact\. Both KL terms are available in closed form, while the reconstruction term is estimated with a single reparameterized sample per section,δi,js=μi,js\+σi,js⊙ϵ\\delta^\{s\}\_\{i,j\}=\\mu^\{s\}\_\{i,j\}\+\\sigma^\{s\}\_\{i,j\}\\odot\\epsilonwithϵ∼𝒩\(0,I\)\\epsilon\\sim\\mathcal\{N\}\(0,I\)\. Since the word likelihood \([2](https://arxiv.org/html/2611.21980#S3.E2)\) depends only on the section logit, the document logit is never sampled and enters the bound only through the two KL terms, which removes one level of Monte Carlo noise\. As in online VB, we optimize the bound on minibatches of documents, recomputing the topicsβ\\betafrom the embeddings via \([1](https://arxiv.org/html/2611.21980#S3.E1)\) at each step\. For all experiments, we fixσd2=σs2=1\\sigma\_\{d\}^\{2\}=\\sigma\_\{s\}^\{2\}=1rather than learning them, so that the model has no free coupling parameter to tune\. Appendix[B](https://arxiv.org/html/2611.21980#A2)ablates this choice on the U\.S\. Trade Statutes and EUR\-Lex:σs2=1\\sigma\_\{s\}^\{2\}=1is near\-optimal on EUR\-Lex, and on the Trade Statutes it is a compromise between looser coupling \(better perplexity\) and tighter coupling \(better topic quality\)\. Appendix[E](https://arxiv.org/html/2611.21980#A5)discusses parameter counts and training time\. Algorithm[1](https://arxiv.org/html/2611.21980#alg1)summarizes the full procedure\. Algorithm 1Topic modeling with hETM1:Initialize topics \{ρ,α1:K\}\\\{\\rho,\\alpha\_\{1:K\}\\\}, encoders \{νd,νs\}\\\{\\nu\_\{d\},\\nu\_\{s\}\\\}, variances σd2,σs2\\sigma\_\{d\}^\{2\},\\sigma\_\{s\}^\{2\} 2:foriteration t=1,2,…t=1,2,\\dotsdo 3:Compute βk=softmax\(ρ⊤αk\)\\beta\_\{k\}=\\mathrm\{softmax\}\(\\rho^\{\\top\}\\alpha\_\{k\}\)for each topic kk 4:Sample a minibatch BBof documents 5:foreach document i∈Bi\\in Bdo 6: \(μid,log\(σid\)2\)←NNd\(x¯i,νd\)\(\\mu\_\{i\}^\{d\},\\log\(\\sigma\_\{i\}^\{d\}\)^\{2\}\)\\leftarrow\\mathrm\{NN\}\_\{d\}\(\\bar\{x\}\_\{i\};\\nu\_\{d\}\) 7:foreach section j=1,…,nij=1,\\dots,n\_\{i\}do 8: \(μi,js,log\(σi,js\)2\)←NNs\(x¯i,j,νs\)\(\\mu\_\{i,j\}^\{s\},\\log\(\\sigma\_\{i,j\}^\{s\}\)^\{2\}\)\\leftarrow\\mathrm\{NN\}\_\{s\}\(\\bar\{x\}\_\{i,j\};\\nu\_\{s\}\) 9: δi,js←μi,js\+σi,js⊙ϵ,ϵ∼𝒩\(0,I\)\\delta\_\{i,j\}^\{s\}\\leftarrow\\mu\_\{i,j\}^\{s\}\+\\sigma\_\{i,j\}^\{s\}\\odot\\epsilon,\\ \\epsilon\\sim\\mathcal\{N\}\(0,I\) 10: θi,js←softmax\(δi,js\)\\theta\_\{i,j\}^\{s\}\\leftarrow\\mathrm\{softmax\}\(\\delta\_\{i,j\}^\{s\}\) 11: p\(xi,j,v\)←\(θi,js\)⊤β⋅,vp\(x\_\{i,j,v\}\)\\leftarrow\(\\theta\_\{i,j\}^\{s\}\)^\{\\top\}\\beta\_\{\\cdot,v\}for each word vv 12:endfor 13: ℒi←\(R\)−\(D\)−\(S\)\\mathcal\{L\}\_\{i\}\\leftarrow\\mathrm\{\(R\)\}\-\\mathrm\{\(D\)\}\-\\mathrm\{\(S\)\}⊳\\trianglerightEqs\. \([5](https://arxiv.org/html/2611.21980#S4.E5)\)–\([7](https://arxiv.org/html/2611.21980#S4.E7)\) 14:endfor 15:Update \{ρ,α1:K,νd,νs\}\\\{\\rho,\\alpha\_\{1:K\},\\nu\_\{d\},\\nu\_\{s\}\\\}\(and optionally σd2,σs2\\sigma\_\{d\}^\{2\},\\sigma\_\{s\}^\{2\}\) by ascending ∑i∈Bℒi\\sum\_\{i\\in B\}\\mathcal\{L\}\_\{i\} 16:endfor ## 5Experimental Setup We evaluate hETM both intrinsically and extrinsically\. In both settings, we compare it against four baseline models: ETM\([Dieng et al\., 2020](https://arxiv.org/html/2611.21980#bib.bib3)\), SawETM\([Duan et al\., 2021](https://arxiv.org/html/2611.21980#bib.bib19)\), TSNTM\([Isonuma et al\., 2020](https://arxiv.org/html/2611.21980#bib.bib22)\), and mlhLDA\([Krstovski et al\., 2016](https://arxiv.org/html/2611.21980#bib.bib7)\)\. The latter couples document\- and section\-level topic distributions through a hierarchical Dirichlet prior; because its topics are full multinomials rather than embeddings, the gap between them isolates the contribution of the embedding\-based likelihood\. We use ETM in two variants: the default version, which infers a single topic distribution over the whole document, denoted ETM\(D\); and a version that treats each section as a separate document and infers a topic distribution per section, denoted ETM\(S\)\. We use four corpora across three domains: legal, finance, and online forums\. ### 5\.1Corpora Across the corpora, the hierarchical document structure is either explicit or implicit\. For every corpus, we build the effective vocabulary with the Mallet package\([McCallum, 2002](https://arxiv.org/html/2611.21980#bib.bib12)\)and apply the same preprocessing, standard for BOW latent variable models of text: we lowercase the text, remove punctuation, and discard any token containing a numeric character\. We also remove stopwords and the 50 most frequent words, along with words that appear in fewer than three documents \(df\(w\)<3df\(w\)<3\) or are shorter than four characters \(\|w\|<4\|w\|<4\)\. Below, we describe each corpus together with its section statistics and effective vocabulary size\. #### U\.S\. trade legislation\. This is a corpus of U\.S\. trade legislation between 1951–2020\. It consists of 576 bills comprising roughly 40k sections, with a heavy\-tailed distribution \(min 1, max 1,828, mean 69\) that makes the section/document distinction sharp; 22% of bills contain a single section\. The effective vocabulary consists of 25,456 tokens\. Figure[1](https://arxiv.org/html/2611.21980#S5.F1)shows an example hETM representation, with 50 topics, of the Omnibus Trade and Competitiveness Act of 1988\. Shown at the top is the inferred topic distribution of the whole document,θd\\theta\_\{d\}, which in hETM serves as the mean around which the section\-topic distributionsθs\\theta\_\{s\}are drawn\. Below it are three example sections \(out of 475\), each with its inferred distributionθs\\theta\_\{s\}and the top eight words for its three most probable topics\. The document distribution concentrates on a few topics, namely tariffs \(T41\), international relations and sanctions \(T18\), and legal enforcement \(T29\), and each section places its mass on the relevant subset, often blending several of them, so thatθd\\theta\_\{d\}is a smoothed aggregate of the sharper section distributions\. Figure 1:hETM representation of the Omnibus Trade and Competitiveness Act of 1988 \(50 topics\): the document distributionθd\\theta\_\{d\}and three example sectionsθs\\theta\_\{s\}with the eight most probable words of their top topics\. #### Edgar\. This is a financial corpus of annual reports \(10\-K filings\) from all U\.S\. publicly traded companies, spanning 1993–2020\([Loukas et al\., 2021](https://arxiv.org/html/2611.21980#bib.bib13)\)\. We use the 2019 train split, which consists of 4,435 10\-K reports\. A 10\-K is a standard form that companies must file annually, and it contains 15 numbered items\. Since not all items contain text, we merge any item with fewer than 50 tokens into the preceding one\. This yields 15,034 sections \(1–4 per document, mean 3\.39\); 1\.24% of filings reduce to a single item after merging\. Because 10\-K reports are long and contain many company\-specific tokens — tickers, product and subsidiary names, personnel — the defaultdf\(w\)<3df\(w\)<3cutoff used for the other corpora leaves a vocabulary dominated by rare, document\-specific noise\. We therefore raise the threshold todf\(w\)≤5df\(w\)\\leq 5on Edgar, yielding a vocabulary of 37,434 words that stays comparable in size to the other three corpora while keeping the words shared across companies\. #### EUR\-Lex\. EUR\-Lex is a corpus of English\-language EU legislation, comprising regulations, directives, and decisions\([Chalkidis et al\., 2019](https://arxiv.org/html/2611.21980#bib.bib14)\)\. We construct a subset of 10,000 documents, each split into its numbered sections, with an average document length of about 439 words\. This yields 74,710 sections, with 4–46 sections per document \(mean 7\.47\); every document has at least four sections, and the effective vocabulary is 14,163 words\. #### 20 Newsgroups\. The 20 Newsgroups collection\([Lang, 1995](https://arxiv.org/html/2611.21980#bib.bib26)\)is a widely used corpus of Usenet posts spread across 20 newsgroups spanning domains such as computing, recreation, science, politics, and religion, some closely related and others unrelated\. Our version contains 18,746 posts, ranging from 624 to 994 per newsgroup \(mean 937\.3\), so no newsgroup collapses to a single section under the implicit hierarchy framing\. We use this collection to illustrate an implicit hierarchy, treating each newsgroup as a document and its posts as sections, with an effective vocabulary of 18,627 words\. ### 5\.2Word Embeddings As with other embedding based topic models, hETM can either learn its own word embeddings jointly with the topics or use pretrained ones\. Because word embeddings specialized for particular domains are now widely available, and our corpora span legal and financial text, we also evaluate hETM with several such pretrained models: LegalBERT\([Chalkidis et al\., 2020](https://arxiv.org/html/2611.21980#bib.bib17)\), InCaseLawBERT\([Paul et al\., 2023](https://arxiv.org/html/2611.21980#bib.bib16)\), FinBERT\([Araci, 2019](https://arxiv.org/html/2611.21980#bib.bib15)\), and ModernBERT\([Warner et al\., 2024](https://arxiv.org/html/2611.21980#bib.bib18)\)\. Since these are contextual models that produce a different vector for each occurrence of a word, whereas hETM requires a single static embedding per vocabulary type, we convert each model into static embeddings by mean\-pooling the contextual representation of each word over the corpus\([Bommasani et al\., 2020](https://arxiv.org/html/2611.21980#bib.bib25)\)\. SawETM and TSNTM do not accept external pretrained embeddings and are run with their default learned embeddings\. ## 6Intrinsic Evaluation We first evaluate model fit using held\-out perplexity\. We randomly choose 10% of the documents as a test set\. For 20 Newsgroups, the held\-out split is over posts within each newsgroup rather than at the parent\-document level, so all 20 newsgroups appear in training\. For each test document, we split each section into two halves\. The first half of each section is used to infer the section\-level topic distribution for section\-based models \(hETM, ETM\(S\), mlhLDA\) and the document\-level topic distribution for document\-based models \(ETM\(D\), SawETM, TSNTM\)\. The inferred distribution is then used to compute perplexity over the words in the remaining half of that section\. Figure[2](https://arxiv.org/html/2611.21980#S6.F2)reports perplexity across the four collections\. For each model and corpus we ran experiments with both self\-trained and pretrained embeddings and report the result for the best performing embedding; Appendix[D](https://arxiv.org/html/2611.21980#A4)lists the chosen embedding per corpus\. Figure 2:Held\-out perplexity \(section completion\) as a function of the number of topicsKKfor ETM\(D\), ETM\(S\), hETM, SawETM, TSNTM, and mlhLDA on the four document corpora\.Across all collections, hETM achieves the lowest held\-out perplexity at nearly every number of topics, with a clear margin over all baselines\. On EUR\-Lex the gap narrows: hETM, SawETM, and TSNTM form a tight cluster at everyKK, with hETM best atK=10K\{=\}10andK=50K\{=\}50\(where it attains the overall lowest EUR\-Lex perplexity\), SawETM marginally lower atK=20K\{=\}20andK=30K\{=\}30, with hETM and SawETM essentially tied atK=100K\{=\}100; hETM has the lowest mean perplexity acrossKKon this corpus\. Among the hierarchical baselines, mlhLDA is the strongest model besides hETM on the U\.S\. Trade Statutes, Edgar, and 20 Newsgroups\. It even edges hETM on Edgar atK=10K\{=\}10, but degrades on EUR\-Lex at largerKK\. The flat embedded models are consistently weakest and stay roughly flat acrossKK, with ETM\(S\) worst on the Trade Statutes and ETM\(D\) worst on Edgar\. ETM\(D\) and ETM\(S\) are trained under the same protocol as hETM; their near\-flatness acrossKKreflects a flat likelihood saturating on sectioned text, not undertraining\. The remaining hierarchical baselines TSNTM and SawETM fall between hETM and mlhLDA, though TSNTM is unstable at largeKK, spiking on Edgar and 20 Newsgroups atK=100K\{=\}100\. Overall, hETM’s perplexity tends to improve or hold asKKgrows, indicating that coupling the section and document topic distributions provides a better fit to these sectioned collections than either flat or topic hierarchy models\. On three of the four corpora, mlhLDA≻\\succflat ETM\(D\)/ETM\(S\) further supports that document–section coupling by itself improves fit, and hETM≻\\succmlhLDA isolates the gain contributed by the embedding\-based likelihood on top of that coupling\. Beyond how well a model fits held\-out text, topic models are also evaluated on the quality of the topics they produce\. Two metrics are standard: topic coherence, computed as NPMI over the top 10 words from document level co\-occurrence\([Lau et al\., 2014](https://arxiv.org/html/2611.21980#bib.bib8)\), and topic diversity, the proportion of unique words among the top words of all topics\([Dieng et al\., 2020](https://arxiv.org/html/2611.21980#bib.bib3)\)\. Following[Dieng et al\. \(2020\)](https://arxiv.org/html/2611.21980#bib.bib3), we combine them into a single measure, topic quality, defined as the product of coherence and diversity, where higher is better\. Figure 3:Topic quality across the four collections, for each model and number of topicsKK\(higher is better\)\.Figure[3](https://arxiv.org/html/2611.21980#S6.F3)reports topic quality across the four collections\. hETM produces the highest quality topics on Edgar at every number of topics\. On the U\.S\. Trade Statutes, mlhLDA is the strongest model forK≤50K\{\\leq\}50and hETM takes the lead atK=100K\{=\}100as mlhLDA’s quality collapses; across the Trade Statutes, mlhLDA and hETM are a clear pair above the remaining baselines\. On EUR\-Lex, SawETM is the strongest competitor and overtakes hETM forK≥20K\{\\geq\}20, remaining high and roughly flat while hETM starts highest atK=10K\{=\}10but declines withKK\. On 20 Newsgroups, the two best models are hETM and the flat ETM\(D\), which trade the lead acrossKK\(ETM\(D\) is highest atK=10K\{=\}10,5050, and100100\), while SawETM and TSNTM are much weaker there\. Elsewhere, the flat and topic\-hierarchy baselines are consistently the weakest, and ETM\(S\) even yields slightly negative quality on EUR\-Lex\. As is typical, topic quality falls asKKgrows for most models\. Tablein Appendix[A](https://arxiv.org/html/2611.21980#A1)illustrates this qualitatively: hETM recovers thematically clean, distinctive topics on the U\.S\. Trade Statutes, while the baselines tend to repeat procedural boilerplate or fuse separate themes\. ## 7Extrinsic Evaluation A key advantage of topic models is that they represent every document in a shared, low\-dimensional topic space, allowing documents with different vocabularies to be compared and used as features in downstream tasks\. Beyond the intrinsic evaluations above, we therefore evaluate hETM extrinsically on a document clustering task, using the inferred document\-topic representations as features and comparing hETM against the baseline models ETM\(D\), ETM\(S\), SawETM, TSNTM, and mlhLDA with k\-means clustering\. For ETM\(S\), which treats each section as a separate document, we aggregate the per\-section topic distributions by mean pooling to obtain a document representation\. We measure cluster quality with the Adjusted Rand Index\([Hubert and Arabie, 1985](https://arxiv.org/html/2611.21980#bib.bib29), ARI;\), a chance\-corrected measure of agreement between the predicted clusters and the ground\-truth labels that is robust to the label imbalance in the U\.S\. Trade Statutes and EUR\-Lex corpora, where unadjusted accuracy can over\-reward predictions that collapse onto the majority class; clustering accuracy and Normalized Mutual Information\([Strehl and Ghosh, 2002](https://arxiv.org/html/2611.21980#bib.bib30), NMI;\)are reported in Appendix[C](https://arxiv.org/html/2611.21980#A3)\. We run the task on three of the four collections, excluding 20 Newsgroups: there the hierarchy is implicit, with each newsgroup treated as a single document, leaving only 20 documents to cluster\. The labels differ by collection: for the U\.S\. Trade Statutes we evaluate clustering on the subset of laws that we assign to one of four trade\-specific categories; for Edgar we use the Standard Industrial Classification \(SIC\) code, which places each company in one of eight sectors; and for EUR\-Lex we use the EuroVoc top\-level classification to assign each law to one of four trade categories\. Table[1](https://arxiv.org/html/2611.21980#S7.T1)reports the best ARI per model and theKKat which it is attained\. Table 1:Document clustering ARI across the three collections \(best per collection in bold\)\. For each model, we select the number of topicsKKthat maximizes ARI; subscripts give that configuration\. hETM is best on the U\.S\. Trade Statutes and Edgar; SawETM is best on EUR\-Lex\.hETM achieves the highest ARI on the U\.S\. Trade Statutes and Edgar datasets, attaining these results at relatively low topic counts \(K=20K\{=\}20andK=10K\{=\}10\)\. In contrast, baseline models reach peak performance only at substantially higherKKvalues, such asK=100K\{=\}100for both TSNTM and ETM\(D\) on Edgar\. On EUR\-Lex, hETM remains competitive but does not outperform all baselines; SawETM achieves the highest ARI, with hETM second\. This is consistent with the intrinsic evaluation, where SawETM also performs best on this corpus\. Among the hierarchical baselines, mlhLDA tracks its intrinsic ranking across corpora, and the hETM–mlhLDA gap again isolates the gain from the embedding\-based likelihood\. No baseline consistently ranks second: ETM\(D\) is weakest on the Trade Statutes, while TSNTM and SawETM alternate elsewhere\. Appendix[C](https://arxiv.org/html/2611.21980#A3)reports accuracy and NMI under the same per\-metric best\-KKselection; all three metrics give the same ranking per corpus, so the ranking is not an artifact of a single metric or of label imbalance\. ## 8Conclusion We introduced hETM, a generative hierarchical embedded topic model for sectioned documents that draws each section’s topic proportions as a Gaussian perturbation of its document’s in logit space\. The coupling keeps the model reparameterizable and yields a closed\-form level\-coupling term fit by amortized variational inference\. Across legal, financial, and online forum corpora, hETM attains the lowest held\-out perplexity at everyKKon the U\.S\. Trade Statutes and 20 Newsgroups, wins four of fiveKKon Edgar \(ceding onlyK=10K\{=\}10to mlhLDA\), and achieves the single lowest EUR\-Lex perplexity point \(K=50K\{=\}50\)\. It also attains the best topic quality on Edgar at everyKKand the best clustering on the U\.S\. Trade Statutes and Edgar, with mlhLDA the strongest topic\-quality competitor at smallKKon the Trade Statutes and SawETM the strongest competitor on EUR\-Lex\. These results support the benefit of coupling section and document topics rather than modeling either level alone\. ## Limitations hETM assumes documents are partitioned into sections; when the partition is implicit, we construct it heuristically\. Like other topic models, hETM relies on bag of words representation and ignores word order within a section\. The hierarchical logistic normal prior approximates rather than exactly mirrors the nested Dirichlet of mlhLDA, since the correspondence holds only in the logit space\. And like other amortized variational models, hETM is subject to the usual amortization gap: a shared inference network cannot reach the per document posterior of mean field variational inference\. The variational posterior factorizes across sections \(Section[4](https://arxiv.org/html/2611.21980#S4)\), so each section posteriorqνs\(δi,js∣xi,j\)q\_\{\\nu\_\{s\}\}\(\\delta^\{s\}\_\{i,j\}\\mid x\_\{i,j\}\)reads only its own section’s word counts\. The coupling between the document and section levels enters the ELBO only through the KL term that pulls the posterior means together\. A variational family of the formq\(δi,js∣xi,j,δid\)q\(\\delta^\{s\}\_\{i,j\}\\mid x\_\{i,j\},\\delta^\{d\}\_\{i\}\), which conditions each section posterior on the document logit, would more faithfully mirror the generative process and is a natural direction for future work\. Although a document is defined as an ordered list of sections, the model treats sections as conditionally independent given the document representation\. Properly exploiting section order would require a sequential prior over section logits and a correspondingly structured inference network\. We leave this to future work\. Finally, our corpora are English and our hierarchies are two levels deep, so extensions to multilingual corpora and to hierarchies with more than two levels remain unevaluated\. ## References - Araci \(2019\)D\. AraciFinBERT: financial sentiment analysis with pre\-trained language models\.arXiv preprint arXiv:1908\.10063\.Cited by:[§5\.2](https://arxiv.org/html/2611.21980#S5.SS2.p1.1)\. - Balikaset al\.\(2016\)G\. Balikas, M\. Amini, and M\. ClauselOn a topic model for sentences\.InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 921–924\.External Links:[Document](https://dx.doi.org/10.1145/2911451.2914714)Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p5.1)\. - Bleiet al\.\(2010\)D\. M\. Blei, T\. L\. Griffiths, and M\. I\. JordanThe nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies\.Journal of the ACM57\(2\),pp\. 1–30\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p1.1)\. - Bleiet al\.\(2003\)D\. M\. Blei, A\. Y\. Ng, and M\. I\. JordanLatent Dirichlet allocation\.Journal of Machine Learning Research3,pp\. 993–1022\.Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p1.1)\. - Bommasaniet al\.\(2020\)R\. Bommasani, K\. Davis, and C\. CardieInterpreting pretrained contextualized representations via reductions to static embeddings\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4758–4781\.Cited by:[Appendix D](https://arxiv.org/html/2611.21980#A4.p2.1),[§5\.2](https://arxiv.org/html/2611.21980#S5.SS2.p1.1)\. - Chalkidiset al\.\(2019\)I\. Chalkidis, E\. Fergadiotis, P\. Malakasiotis, and I\. AndroutsopoulosLarge\-scale multi\-label text classification on EU legislation\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 6314–6322\.External Links:[Link](https://aclanthology.org/P19-1636/),[Document](https://dx.doi.org/10.18653/v1/P19-1636)Cited by:[§5\.1](https://arxiv.org/html/2611.21980#S5.SS1.SSS0.Px3.p1.1)\. - Chalkidiset al\.\(2020\)I\. Chalkidis, M\. Fergadiotis, P\. Malakasiotis, N\. Aletras, and I\. AndroutsopoulosLEGAL\-BERT: the muppets straight out of law school\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 2898–2904\.Cited by:[§5\.2](https://arxiv.org/html/2611.21980#S5.SS2.p1.1)\. - Daset al\.\(2015\)R\. Das, M\. Zaheer, and C\. DyerGaussian LDA for topic models with word embeddings\.InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 795–804\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p3.1)\. - Dienget al\.\(2020\)A\. B\. Dieng, F\. J\. R\. Ruiz, and D\. M\. BleiTopic modeling in embedding spaces\.Transactions of the Association for Computational Linguistics8,pp\. 439–453\.Cited by:[Appendix D](https://arxiv.org/html/2611.21980#A4.p1.1),[§1](https://arxiv.org/html/2611.21980#S1.p1.1),[§1](https://arxiv.org/html/2611.21980#S1.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p3.1),[§5](https://arxiv.org/html/2611.21980#S5.p1.1),[§6](https://arxiv.org/html/2611.21980#S6.p5.1)\. - Duanet al\.\(2021\)Z\. Duan, D\. Wang, B\. Chen, C\. Wang, W\. Chen, Y\. Li, J\. Ren, and M\. ZhouSawtooth factorial topic embeddings guided gamma belief network\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),PMLR, Vol\.139,pp\. 2903–2913\.Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p4.1),[§5](https://arxiv.org/html/2611.21980#S5.p1.1)\. - Hoffmanet al\.\(2010\)M\. D\. Hoffman, D\. M\. Blei, and F\. BachOnline learning for latent Dirichlet allocation\.InAdvances in Neural Information Processing Systems \(NIPS\),pp\. 856–864\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p2.1)\. - Hubert and Arabie \(1985\)L\. Hubert and P\. ArabieComparing partitions\.Journal of Classification2\(1\),pp\. 193–218\.Cited by:[§7](https://arxiv.org/html/2611.21980#S7.p1.1)\. - Isonumaet al\.\(2020\)M\. Isonuma, J\. Mori, D\. Bollegala, and I\. SakataTree\-structured neural topic model\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 800–806\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p4.1),[§5](https://arxiv.org/html/2611.21980#S5.p1.1)\. - Jianget al\.\(2019\)H\. Jiang, R\. Zhou, L\. Zhang, H\. Wang, and Y\. ZhangSentence level topic models for associated topics extraction\.World Wide Web22\(6\),pp\. 2545–2560\.External Links:[Document](https://dx.doi.org/10.1007/s11280-018-0639-1)Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p5.1)\. - Kimet al\.\(2013\)D\. Kim, G\. Voelker, and L\. K\. SaulA variational approximation for topic modeling of hierarchical corpora\.InProceedings of the 30th International Conference on Machine Learning \(ICML\),pp\. 55–63\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p1.1)\. - Kingma and Welling \(2014\)D\. P\. Kingma and M\. WellingAuto\-encoding variational Bayes\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p5.1)\. - Krstovskiet al\.\(2016\)K\. Krstovski, D\. A\. Smith, and M\. J\. KurtzOnline multilingual topic models with multi\-level hyperpriors\.InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),pp\. 454–459\.Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p2.1),[§5](https://arxiv.org/html/2611.21980#S5.p1.1)\. - Lang \(1995\)K\. LangNewsweeder: learning to filter netnews\.InProceedings of the 12th International Conference on Machine Learning \(ICML\),pp\. 331–339\.Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p2.1),[§5\.1](https://arxiv.org/html/2611.21980#S5.SS1.SSS0.Px4.p1.1)\. - Lauet al\.\(2014\)J\. H\. Lau, D\. Newman, and T\. BaldwinMachine reading tea leaves: automatically evaluating topic coherence and topic model quality\.InProceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),pp\. 530–539\.Cited by:[§6](https://arxiv.org/html/2611.21980#S6.p5.1)\. - Liet al\.\(2016\)S\. Li, T\. Chua, J\. Zhu, and C\. MiaoGenerative topic embedding: a continuous representation of documents\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Berlin, Germany,pp\. 666–675\.External Links:[Link](https://aclanthology.org/P16-1063/)Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p3.1)\. - Liet al\.\(2017\)S\. Li, Y\. Zhang, R\. Pan, M\. Mao, and Y\. YangRecurrent attentional topic model\.InProceedings of the Thirty\-First AAAI Conference on Artificial Intelligence,pp\. 3223–3229\.External Links:[Document](https://dx.doi.org/10.1609/AAAI.V31I1.10972)Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p5.1)\. - Li and McCallum \(2006\)W\. Li and A\. McCallumPachinko allocation: DAG\-structured mixture models of topic correlations\.InProceedings of the 23rd International Conference on Machine Learning \(ICML\),pp\. 577–584\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p1.1)\. - Loukaset al\.\(2021\)L\. Loukas, M\. Fergadiotis, I\. Androutsopoulos, and P\. MalakasiotisEDGAR\-CORPUS: billions of tokens make the world go round\.InProceedings of the Third Workshop on Economics and Natural Language Processing,U\. Hahn, V\. Hoste, and A\. Stent \(Eds\.\),Punta Cana, Dominican Republic,pp\. 13–18\.External Links:[Link](https://aclanthology.org/2021.econlp-1.2/),[Document](https://dx.doi.org/10.18653/v1/2021.econlp-1.2)Cited by:[§5\.1](https://arxiv.org/html/2611.21980#S5.SS1.SSS0.Px2.p1.1)\. - McCallum \(2002\)A\. K\. McCallumMALLET: a machine learning for language toolkit\.Note:[http://mallet\.cs\.umass\.edu](http://mallet.cs.umass.edu/)Cited by:[§5\.1](https://arxiv.org/html/2611.21980#S5.SS1.p1.1)\. - Miaoet al\.\(2016\)Y\. Miao, L\. Yu, and P\. BlunsomNeural variational inference for text processing\.InProceedings of the 33rd International Conference on Machine Learning \(ICML\),pp\. 1727–1736\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p3.1)\. - Nguyenet al\.\(2025\)T\. Nguyen, D\. Doan, H\. Luong, and L\. Ngo VanSub\-document neural topic model\.Knowledge\-Based Systems331,pp\. 114762\.External Links:[Document](https://dx.doi.org/10.1016/j.knosys.2025.114762)Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p5.1)\. - Paulet al\.\(2023\)S\. Paul, A\. Mandal, P\. Goyal, and S\. GhoshPre\-trained language models for the legal domain: a case study on Indian law\.InProceedings of the 19th International Conference on Artificial Intelligence and Law \(ICAIL\),Note:arXiv:2209\.06049Cited by:[§5\.2](https://arxiv.org/html/2611.21980#S5.SS2.p1.1)\. - Srivastava and Sutton \(2017\)A\. Srivastava and C\. SuttonAutoencoding variational inference for topic models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p3.1)\. - Strehl and Ghosh \(2002\)A\. Strehl and J\. GhoshCluster ensembles — a knowledge reuse framework for combining multiple partitions\.Journal of Machine Learning Research3,pp\. 583–617\.Cited by:[Appendix C](https://arxiv.org/html/2611.21980#A3.p1.1),[§7](https://arxiv.org/html/2611.21980#S7.p1.1)\. - Viegaset al\.\(2025\)F\. Viegas, A\. Pereira, W\. Cunha, C\. França, C\. Andrade, E\. Tuler, L\. Rocha, and M\. A\. GonçalvesExploiting contextual embeddings in hierarchical topic modeling and investigating the limits of the current evaluation metrics\.Computational Linguistics51\(3\),pp\. 843–883\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p4.1)\. - Wallachet al\.\(2009\)H\. M\. Wallach, D\. Mimno, and A\. McCallumRethinking LDA: why priors matter\.InAdvances in Neural Information Processing Systems \(NIPS\),pp\. 1973–1981\.Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p1.1)\. - Wanget al\.\(2009\)D\. Wang, S\. Zhu, T\. Li, and Y\. GongMulti\-document summarization using sentence\-based topic models\.InProceedings of the ACL\-IJCNLP 2009 Conference Short Papers,pp\. 297–300\.Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p5.1)\. - Warneret al\.\(2024\)B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. PoliSmarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.arXiv preprint arXiv:2412\.13663\.Cited by:[§5\.2](https://arxiv.org/html/2611.21980#S5.SS2.p1.1)\. - Wuet al\.\(2024a\)X\. Wu, T\. Nguyen, and A\. T\. LuuA survey on neural topic models: methods, applications, and challenges\.Artificial Intelligence Review57\(2\)\.Note:arXiv:2401\.15351Cited by:[§2](https://arxiv.org/html/2611.21980#S2.p4.1)\. - Wuet al\.\(2024b\)X\. Wu, F\. Pan, T\. Nguyen, Y\. Feng, C\. Liu, C\. Nguyen, and A\. T\. LuuOn the affinity, rationality, and diversity of hierarchical topic modeling\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38\.Note:arXiv:2401\.14113Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p6.1)\. - Xuet al\.\(2022\)Y\. Xu, D\. Wang, B\. Chen, R\. Lu, Z\. Duan, and M\. ZhouHyperMiner: topic taxonomy mining with hyperbolic embedding\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35\.Note:arXiv:2210\.10625Cited by:[§1](https://arxiv.org/html/2611.21980#S1.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p4.1),[§2](https://arxiv.org/html/2611.21980#S2.p6.1)\. ## Appendix AExample Topics Tableshows example topics recovered by each model on the U\.S\. Trade Statutes corpus, grouped by theme, with coherence \(C\) and diversity \(D\) as defined in Section[6](https://arxiv.org/html/2611.21980#S6)\. hETM recovers thematically clean, distinctive topics, whereas the baselines either contaminate the theme with procedural boilerplate \(ETM\(D\)\), fragment or fuse it \(TSNTM, SawETM\), or produce low\-coherence mixtures \(ETM\(S\)\)\. mlhLDA atK=50K=50recovers none of these themes, instead splitting the vocabulary into narrow product clusters \(for example anti\-doping and licensing, plant pathogens, and chemical descriptions\), and only produces a clean agriculture topic atK=100K=100— an illustration of how its free categorical topic\-word distributions fragment the topic space\. ## Appendix BSection Variance Ablation The section varianceσs2\\sigma\_\{s\}^\{2\}in hETM controls how tightly each section’s topic distribution is pulled toward its document’s\. Two limiting cases bracket its role:σs2→0\\sigma\_\{s\}^\{2\}\\to 0makes every section share its document’s mixture, reducing the reconstruction term to that of ETM\(D\), whileσs2→∞\\sigma\_\{s\}^\{2\}\\to\\inftyeliminates this coupling entirely, so sections are fit independently, closer to ETM\(S\)\. To check that the coupling does real work in between, we sweepσs2∈\{0\.1,1,10\}\\sigma\_\{s\}^\{2\}\\in\\\{0\.1,\\,1,\\,10\\\}\(small, default, large\) and retrain hETM atK∈\{10,20,30,50,100\}K\\in\\\{10,20,30,50,100\\\}on two corpora chosen for their contrasting section topic structure: the U\.S\. Trade Statutes, where sections of a single bill often address different subtopics, and EUR\-Lex, where sections of a single regulation tend to stay on a single theme\. Figure 4:Held\-out perplexity of hETM as a function of the number of topicsKKforσs2∈\{0\.1,1,10\}\\sigma\_\{s\}^\{2\}\\in\\\{0\.1,\\,1,\\,10\\\}on the U\.S\. Trade Statutes and EUR\-Lex\. The defaultσs2=1\\sigma\_\{s\}^\{2\}=1used throughout the paper is in green\.Figure 5:Topic quality \(coherence multiplied by diversity\) for the same ablation\. Higher is better\.Figures[4](https://arxiv.org/html/2611.21980#A2.F4)and[5](https://arxiv.org/html/2611.21980#A2.F5)show the sweep\. The two corpora behave differently, and the pattern lines up with how much section topic variation each contains\. On EUR\-Lex the three curves stay close together on both metrics, and the defaultσs2=1\\sigma\_\{s\}^\{2\}=1gives the lowest perplexity at everyKKand the highest topic quality at mostKK\. On the U\.S\. Trade Statutes the three curves fan out, and the sweep exposes a trade off: the loose extremeσs2=10\\sigma\_\{s\}^\{2\}=10gives the lowest perplexity atK≥20K\\geq 20while the tight extremeσs2=0\.1\\sigma\_\{s\}^\{2\}=0\.1gives the highest topic quality atK≥20K\\geq 20; the defaultσs2=1\\sigma\_\{s\}^\{2\}=1sits between the two\. This is the regime hETM is designed for: because sections of a single bill address substantially different subtopics, letting each section drift \(σs2=10\\sigma\_\{s\}^\{2\}=10\) throws away the shared document theme and collapses topic quality at largeKK, while tightening too far \(σs2=0\.1\\sigma\_\{s\}^\{2\}=0\.1\) forces sections into the document’s average mixture and hurts perplexity\. The trade off is itself informative\. On corpora where sections genuinely cover different subtopics, the choice ofσs2\\sigma\_\{s\}^\{2\}trades section level fit against collection level topic coherence, and no single value dominates\. On corpora where section level variation is small, this trade off collapses and the default suffices\. In the main paper experiments we fixσs2=1\\sigma\_\{s\}^\{2\}=1as a balanced default\. Algorithm[1](https://arxiv.org/html/2611.21980#alg1)notes that the variances can optionally be learned, but we did not do so in Sections[6](https://arxiv.org/html/2611.21980#S6)and[7](https://arxiv.org/html/2611.21980#S7), so this ablation isolates a single hyperparameter\. We did not rerun the full sweep on Edgar or 20 Newsgroups\. The pattern on the two corpora shown here, in which sensitivity toσs2\\sigma\_\{s\}^\{2\}scales with the amount of section topic variation in a corpus, predicts that these two sit between the extremes shown; we leave empirical verification to future work\. ## Appendix CClustering Accuracy and NMI For completeness, and to confirm that the ARI ranking in Table[1](https://arxiv.org/html/2611.21980#S7.T1)is not an artifact of a single metric, Table[3](https://arxiv.org/html/2611.21980#A3.T3)and Table[4](https://arxiv.org/html/2611.21980#A3.T4)report clustering accuracy and Normalized Mutual Information \(NMI\)\([Strehl and Ghosh, 2002](https://arxiv.org/html/2611.21980#bib.bib30)\)for the same models and corpora\. For each table and each model we select the number of topicsKKthat maximizes the reported metric and give it as a subscript, as in the main\-body ARI table\. Table 3:Document clustering accuracy across the three collections \(best per collection in bold\)\. For each model we select the number of topicsKKthat maximizes accuracy; subscripts give that configuration\.Table 4:Document clustering Normalized Mutual Information \(NMI\) across the three collections \(best per collection in bold\)\. For each model we select the number of topicsKKthat maximizes NMI; subscripts give that configuration\. ## Appendix DTraining Configuration and Reproducibility In this appendix we provide the training configuration used for all hETM and ETM results in the paper, along with the pretrained\-embedding selection for hETM\. ETM is the standard ETM implementation of[Dieng et al\. \(2020\)](https://arxiv.org/html/2611.21980#bib.bib3)with its default settings\. hETM uses two separate variational encoders, one at the document level and one at the section level, each with two fully connected hidden layers of 800 units and ReLU activations, taking normalized bag\-of\-words inputs and parameterizing diagonal Gaussian posteriors\. The document and section encoders share the global ETM topic\-word decoder, with topic embeddings of the same dimension as the word embeddings \(768 by default, 512 when LegalBERT\-small is used\)\. All models are trained with Adam, learning rate 0\.005, weight decay1\.2×10−61\.2\\times 10^\{\-6\}, and no dropout, with early stopping\. hETM uses minibatches of 16 parent documents and processes sections in chunks of up to 64 \(USTS, Edgar\) or 128 \(EUR\-Lex\) per document; ETM uses minibatches of 32 documents on USTS and Edgar and 64 on EUR\-Lex\. We use a 90/10 train/test split taken at the parent\-document level with preparation seed 42; training uses seed 2019\. Pretrained word embeddings \(LegalBERT, InCaseLawBERT, FinBERT, ModernBERT; dimensions 768 or 512\) are kept frozen during optimization\. The learned configuration initializes embeddings randomly and trains them jointly with the topic model\. Pretrained word vectors are obtained following[Bommasani et al\. \(2020\)](https://arxiv.org/html/2611.21980#bib.bib25): for each vocabulary word, we collect its contextual representation at every occurrence in the corpus and average these vectors to a single static embedding per type\. Section[5\.2](https://arxiv.org/html/2611.21980#S5.SS2)introduces six embedding configurations for hETM \(LegalBERT\-base, LegalBERT\-small, InCaseLawBERT, FinBERT, ModernBERT, and learned\)\. We train hETM once per configuration and the main text reports the configuration with the lowest held\-out perplexity\. Table[5](https://arxiv.org/html/2611.21980#A4.T5)lists the winning configuration on each corpus\. Table 5:Pretrained \(or learned\) word embedding configuration that achieved the lowest held\-out perplexity for hETM on each corpus\. Six configurations are compared; see Section[5\.2](https://arxiv.org/html/2611.21980#S5.SS2)for the configuration details\.On three of the four corpora the learned embedding wins; on EUR\-Lex LegalBERT\-base wins\. We attribute this to corpus size and vocabulary coverage: EUR\-Lex is our largest corpus and its legal vocabulary overlaps substantially with LegalBERT’s pretraining, so the pretrained embedding wins over a learned one even after mean\-pooling its subword pieces\. On the smaller corpora, and on 20 Newsgroups whose informal vocabulary matches none of our pretrained models, a learned embedding has more freedom to adapt to the actual vocabulary and outperforms every pretrained option we tested\. ## Appendix EParameter Counts and Training Time hETM uses a second variational encoder for the section level, which roughly doubles the encoder parameter count relative to ETM\. The topic word decoder is shared across the two levels, so the full hETM parameter count stays below twice that of ETM at the same vocabulary and topic count\. Training time per epoch grows with the mean number of sections per document: hETM is under two times slower than ETM on Edgar \(mean 3\.4 sections per document\) and about five times slower on the U\.S\. Trade Statutes \(mean 69 sections per document\)\. At inference time hETM applies both encoders, so inference latency per document is roughly twice ETM’s\. All runs reported in the main text, across all four corpora and all topic counts, fit within 24 hours of wall clock time on a single Nvidia RTX 2080TI GPU\.
Similar Articles
CobwebTM: Probabilistic Concept Formation for Lifelong and Hierarchical Topic Modeling
CobwebTM is a low-parameter lifelong hierarchical topic modeling approach that adapts the Cobweb algorithm to continuous document embeddings, enabling unsupervised topic discovery and dynamic hierarchical organization without predefining topic counts. The method combines incremental symbolic concept formation with pretrained representations to achieve strong topic coherence while avoiding catastrophic forgetting.
H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification
The paper proposes H2EDL, a hyper evidential deep learning model for hierarchical classification that captures uncertainty at multiple levels of a label hierarchy, reducing calibration error and improving the preservation of coarse categories in fine-grained recognition tasks.
HCOE: Hyperbolic Clinical Ontology Embeddings from Biomedical Language Models
This paper introduces Hyperbolic Clinical Ontology Embeddings (HCOE), a method that maps biomedical language model embeddings into a hyperbolic space to better capture medical code hierarchies, outperforming existing methods on clinical tasks.
TETHER: Training-Free, Elastic, ToC-Anchored Hierarchy Extraction and Recovery in Design-Driven Long Documents
The paper introduces TETHER, a training-free, two-stage neuro-symbolic framework that uses the Table of Contents as a structural prior with a Large Multimodal Model to extract and recover document hierarchies in long design-driven documents, achieving SOTA zero-shot results with 90% token reduction over end-to-end baselines.
@v0xium: LLM Inference Engineering: Embedding Models Explained 1. An Embedding Model (EM) converts a chunk of text, or any other…
This article explains embedding models and their role in LLM inference, covering architectures, traffic profiles, and optimization techniques like quantization.