Automatic Thematic Indexing of Large Literary Corpora: A Machine Learning Approach to Voltaire's Complete Works

# Automatic Thematic Indexing of Large Literary Corpora: A Machine Learning Approach to Voltaire's Complete Works
Source: [https://arxiv.org/html/2607.09316](https://arxiv.org/html/2607.09316)
\[2\]Gillian Pink

###### Abstract

Thematic indexing — the practice of assigning structured conceptual labels to sections of text — is essential to scholarly access in large\-scale literary and historical editions, yet it remains a largely manual, labour\-intensive process\. This paper explores the application of machine learning to automatic thematic indexing, using two substantial sub\-corpora of theComplete Works of Voltaireas a test case: theEssai sur les mœurs et l'esprit des nationsand theQuestions sur l'Encyclopédie\. The task is framed as a multi\-label classification problem, in which a model must assign the set of index entries that a professional indexer would apply to a given page of text\. We compare a range of approaches — from encoder\-based models with classification heads to generative large language models \(LLMs\) fine\-tuned via Low\-Rank Adaptation \(LoRA\) — spanning model sizes from approximately 3 to 120 billion parameters\. Our best\-performing model, from the Mistral family in a 4\-bit quantised configuration, achieves F1 scores of up to 0\.67; we argue that these figures represent lower bounds, given the inherent subjectivity of professional indexing and the frequency with which model predictions prove semantically valid despite diverging from the print index\. We further evaluate cross\-corpus generalisation and conduct a detailed qualitative analysis of model behaviour on literary and rhetorical features of the source texts that prove particularly resistant to automated treatment\. Our findings have implications for the broader challenge of providing structured thematic access to large\-scale literary and historical corpora\.

## 1Introduction

One of the founding assumptions of digital editions, and of digitised text more generally, is that they are inherentlyaccessible— not only because digital channels have dramatically expanded their availability, but because they are, in a word, searchable\. Yet the equation of searchability with access deserves scrutiny\. Full text search, for all its power, offers a surprisingly narrow mode of engagement with a textual corpus: it presupposes that the reader already knows what terms to look for\. The user arrives with a query in hand; the text responds or it does not\. Discovery, in any richer sense, is left largely to chance\[whitelaw2012towards\]\.

Traditional scholarly publishing, by contrast, has long relied on a rather different instrument\. A well\-constructed analytical index is not a concordance, nor a finding aid in any merely mechanical sense: it is an interpretative map of a text's contents, compiled by a professional indexer, often with specialist knowledge, who reads the work in its entirety, identifies key concepts and headwords, and progressively refines a structure of categories and sub\-categories as the work takes shape\. The result is qualitatively different from what can be achieved using a search function — a structured invitation to lateral reading, browsing, conceptual navigation, and the kind of serendipitous discovery that no keyword query can reliably produce\[stephen2009print,hjorland2018indexing\]\.

For large corpora, however, manual indexing quickly becomes either prohibitively expensive or simply impractical\. TheŒuvres complètes de Voltaire / Complete Works of Voltaire\[voltaire\_ocv\]presents an instructive case in point\. Published by the Voltaire Foundation over fifty\-five years, the edition runs to 205 print volumes and approximately fifteen million words, encompassing the full range of Voltaire's literary, philosophical, historical, and polemical output\[cronk2023rediscovering\]\. The sheer scale of the undertaking creates a genuine problem of access: most individual works within the corpus remain unknown to non\-specialists, and even lifelong readers of Voltaire can struggle to navigate the full breadth of his writings without structured aids to guide them\. Now that the editorial project is complete and a state\-of\-the\-art online edition has been launched,111[https://www\.ov\-vf\.com/](https://www.ov-vf.com/)the question of how to make this body of text genuinely accessible, not merely searchable, has become newly relevant\[Barker\_Cronk\_Roe\_2026\]\.

Automatic indexing — broadly understood as text categorisation or classification — offers a potential solution to this challenge\. It is a well\-established, if technically demanding, task within the broader field of supervised machine learning\[sebastiani2002,dhar2021text\]\. In its most common formulation, a classification algorithm is trained to recognise the characteristics associated with target index terms, drawing usually on measures of lexical or semantic similarity within a manually indexed document set; it then applies what it has learned to assign terms to previously unseen texts\. The approach is well suited to domains where training data is abundant and documents are relatively homogeneous in length, register, and subject matter\.

The application of such methods to more complex texts, however, is less straightforward\[duncan2022, p\. 250\], and literary and historical corpora give rise to particular difficulties\. Collections like the OCV inhabit an unusually uneven information space: individual documents range from a handful of lines of verse to several hundreds of pages of historiographical prose, and the semantic range of the corpus as a whole — spanningbelles\-lettres, philosophical polemic, private correspondence, physics, and biblical criticism — far exceeds that of the standard benchmarking corpora on which most text classification systems have been developed and evaluated\. Newswire, patent filings, and biomedical abstracts, the workhorses of the field, are tidy by comparison\.

This paper presents a series of experiments in automatic thematic indexing applied to two substantial sub\-corpora of the OCV: the nine\-volumeEssai sur les mœurs et l'esprit des nations\(EM\) and the eight\-volumeQuestions sur l'Encyclopédie\(QE\)\. Both works are the subject of detailed analytical indexes produced as part of the print edition, which we use as ground\-truth training data for a multi\-label classification task\. The problem is framed as follows: given a page of text drawn from either work \(with the unit of the page maintained to preserve compatibility and comparability with the print indexes\), can a machine learning model correctly assign the set of thematic index entries that a professional indexer would have applied?

We compare a range of approaches, from encoder\-based models with classification heads to generative large language models \(LLMs\) fine\-tuned via Low\-Rank Adaptation \(LoRA\)\[hu2022lora\]\. Our experiments span models at considerably different scales — from approximately five billion to over 120 billion parameters — and systematically assess the effect of label frequency, dataset composition, and architectural choice on classification performance\. We further evaluate the capacity of models trained on one work to generalise to the other, providing an initial measure of out\-of\-distribution robustness\.

Our findings suggest that fine\-tuned generative LLMs — and in particular the Mistral\-Small\-3\.2\-24B model in a 4\-bit quantised configuration — offer the most promising results for this task, achieving F1 scores of up to 0\.67 on the QE dataset when low\-frequency labels are filtered\. The results nonetheless underscore the difficulty of the problem: the heavily skewed label distributions characteristic of humanistic indexes, combined with the semantic richness and generic diversity of the source texts, pose challenges that current models can only partially address\. We discuss these limitations and reflect on their implications for the broader ambition of automatically indexing large\-scale literary and historical corpora\.

The paper proceeds as follows\. Section[2](https://arxiv.org/html/2607.09316#S2)situates the work within related research on automatic text classification, multi\-label learning, and computational indexing\. Section[3](https://arxiv.org/html/2607.09316#S3)describes the datasets and preprocessing pipeline\. Section[4](https://arxiv.org/html/2607.09316#S4)details our methodology and presents experimental results across three phases of model comparison\. Section[5](https://arxiv.org/html/2607.09316#S5)discusses the implications of our findings for humanistic text collections more broadly, and Section[6](https://arxiv.org/html/2607.09316#S6)concludes\.

## 2Related Work

The task of automatically generating thematic indexes for literary and historical texts sits at the intersection of several active research areas: automatic text classification, multi\-label learning, computational indexing, and the application of natural language processing \(NLP\) to historical text collections\. We review each in turn, situating our contribution within the current state of the art\.

### 2\.1Automatic Text Classification

Text classification — the assignment of predefined category labels to documents — has been a central problem in information retrieval and natural language processing for several decades\[kadhim2019survey\]\. The field's methodological foundations are surveyed comprehensively in\[sebastiani2002\], which covers document representation, dimensionality reduction, classifier construction, and evaluation, and remains an essential point of reference\. The dominant paradigm through the early 2000s relied on bag\-of\-words representations combined with classifiers such as support vector machines \(SVMs\), naïve Bayes, and k\-nearest neighbours\[manning2008\]\. These approaches achieved strong results on standard benchmarks — Reuters newswire, 20 Newsgroups — but were typically designed and evaluated for single\-label or binary classification settings, where each document belongs to one category or must be distinguished from one other\.

The advent of deep learning brought a fundamental shift in text classification methodology\. Convolutional and recurrent neural networks demonstrated the capacity to learn hierarchical feature representations directly from raw text, dispensing with hand\-crafted feature engineering in favour of end\-to\-end training\[kim2014,zhang2015\]\. The more consequential transformation came with the introduction of transformer\-based pre\-trained language models: beginning with BERT\[devlin2019\], a new paradigm emerged in which large models pre\-trained on massive unlabelled corpora are subsequently fine\-tuned on task\-specific data\. This transfer learning approach has yielded state\-of\-the\-art results across a wide range of NLP benchmarks, text classification prominently among them\[sun2019,howard2018\]\.

### 2\.2Multi\-Label Classification

In many real\-world classification scenarios, however, including the thematic indexing task addressed in this paper, each instance may be associated with multiple labels simultaneously\. Thus, multi\-label classification introduces distinctive challenges beyond those of standard single\-label settings: label co\-occurrence dependencies, class imbalance, and the combinatorial explosion of possible label sets\.\[tsoumakas2007\]provide an influential overview of the field, distinguishing between problem transformation methods, which reduce multi\-label classification to a set of binary classification tasks, and algorithm adaptation methods, which modify existing classifiers to handle multiple labels natively\. A more recent and comprehensive treatment is offered by\[gibaja2015\], covering evaluation measures, label correlation modelling, and scalability considerations\.

A persistent challenge in multi\-label classification is the long\-tailed distribution of labels: in most real\-world datasets, a small number of labels account for a disproportionate share of instances while the majority remain rare\[huang2021balancing\]\. This problem is particularly acute in humanistic indexing, where the analytical vocabulary is both extensive and finely granular\. Our own datasets illustrate the point clearly: 75% of labels in the QE index appear fewer than seven times, and 51% of EM labels fewer than three times\. Strategies for addressing label imbalance include weighted loss functions, label\-aware sampling, and the selective removal of low\-frequency labels — all approaches explored in our experiments\.

### 2\.3Automatic Indexing

The specific task of generating back\-of\-book or subject indexes automatically has received comparatively less attention than general text classification, despite its practical utility for publishers, libraries, and digital archives\. The foundational framework for the field is laid out in\[lancaster2003\], which distinguishes between assigned indexing — the application of terms from a controlled vocabulary — and derived indexing — the extraction of terms from the text itself — and remains the essential starting point for any systematic treatment of the problem\.\[csomai2008\]investigated automatic back\-of\-book indexing using both unsupervised methods \(TF\-IDF, KL divergence\) and supervised learning with linguistically motivated features, demonstrating that the task requires attention to both named entities and conceptual terms — a distinction with particular salience for the kind of thematic indexing we address here\. More recently,\[golub2019\]reviewed the state of automated subject indexing, examining the relationship between automatic classification and controlled vocabularies such as thesauri and classification schemes, and identifying the persistent gap between system\-generated and professionally curated indexes\.

Our work departs from much of this prior literature in a fundamental respect: we do not treat indexing as a term extraction problem\. Rather than identifying salient phrases within the text and elevating them to index status, we formulate the task as a closed\-vocabulary multi\-label classification problem, in which the full set of possible index entries is defined in advance by the existing print indexes and the model must learn to assign the correct subset to each page\. This formulation aligns more closely with subject indexing in the library science tradition than with back\-of\-book indexing as conventionally understood, and it allows us to leverage the carefully curated analytical indexes of the OCV simultaneously as training signal and evaluation standard\.

### 2\.4NLP for Historical Texts

The application of NLP methods to historical texts has grown substantially in recent years, driven by the increasing availability of digitised collections and by advances in language model capabilities\[piotrowski2012natural\]\. A significant strand of this work has focused on named entity recognition \(NER\) and linking in historical documents\. The HIPE shared tasks\[ehrmann2020,ehrmann2022\], organised within the CLEF evaluation forum, established multilingual benchmarks for NER on historical newspapers and foregrounded a set of challenges specific to such texts: OCR noise, archaic orthography, evolving named entities, and the scarcity of large annotated training sets\. These challenges are broadly analogous to those we encounter in working with eighteenth\-century French literary prose, where spelling conventions, editorial apparatus, and the density of historical and geographical reference all complicate the classification task — though our setting adds a further layer of difficulty in the form of the long\-tailed, semantically fine\-grained label space described above\.

Beyond NER, classification and clustering methods have been applied to a range of historical text analysis tasks\. Genre classification has received sustained attention, with\[underwood2019\]demonstrating that machine learning models can track the long\-term evolution of literary genres across large diachronic corpora\. Topic modelling has similarly been applied productively to historical collections\[blei2003,roe2016discourses,schoch2017\], though its sensitivity to the orthographic instability and lexical heterogeneity of pre\-modern texts has prompted ongoing methodological reflection\. Authorship attribution represents a further well\-established classification task with a substantial literature in historical and early modern settings\[koppel2009computational\], while historical text normalisation — the standardisation of archaic and variant spellings as a preprocessing step — has emerged as a critical enabler of downstream classification performance\[bollmann2019large\]\. These studies collectively demonstrate that while standard NLP pipelines can be productively applied to historical texts, some form of domain adaptation — whether through specialised training data, fine\-tuning, or task\-specific architectural choices — is typically necessary to achieve acceptable performance\. Our own task inherits all of these difficulties while adding one further feature: the label space we work with is not a general\-purpose taxonomy but an analytically refined scholarly index, the categories of which reflect interpretative judgements that no off\-the\-shelf model has been trained to replicate\.

### 2\.5LLM Fine\-Tuning for Classification

The emergence of large language models with tens or hundreds of billions of parameters has opened new avenues for text classification, particularly through instruction tuning and parameter\-efficient fine\-tuning methods\. The LoRA framework\[hu2022lora\]introduced the concept of injecting trainable low\-rank decomposition matrices into the layers of a frozen pre\-trained model, dramatically reducing both the number of trainable parameters and the memory footprint of fine\-tuning without sacrificing downstream task performance\. QLoRA\[dettmers2023\]extended this approach by combining LoRA with 4\-bit quantisation, making it possible to fine\-tune models with up to 65 billion parameters on consumer\-grade hardware — a development that has substantially lowered the barrier to entry for research groups without access to large\-scale GPU infrastructure\.

These techniques have been widely adopted for adapting LLMs to domain\-specific tasks, including multi\-label classification\[ma2025large\]\. In our experiments, we employ LoRA\-based fine\-tuning of quantised generative models, treating the models as text decoders that produce the predicted label set directly as a structured text string — a generative formulation that contrasts with the more conventional encoder\-plus\-classifier\-head architecture\. Our comparative results suggest that, for the particular challenges of thematic indexing — large label spaces, low\-frequency labels, semantically rich and generically diverse texts — this approach offers meaningful advantages, a finding we return to in the discussion\.

The present study represents, to our knowledge, one of the first systematic attempts to apply LLM fine\-tuning to the automatic generation of analytical indexes for a large\-scale literary and historical corpus\. While the individual components of our approach — multi\-label classification, LoRA\-based fine\-tuning, historical text processing — are each well established in their respective literatures, their combination in the service of scholarly indexing has not previously been attempted\. The results offer both practical guidance for digital edition projects confronting the accessibility problem outlined above, and methodological contributions to the broader field of computational text analysis in the humanities\.

## 3Dataset and Preprocessing

The two texts selected for this study are Voltaire'sEssai sur les mœurs et l'esprit des nations\(EM\) andQuestions sur l'Encyclopédie\(QE\), both of which are represented in the OCV by detailed analytical indexes produced as part of the print edition\. For EM, seven volumes were used — numbered 22, 23, 24, 25, 26A, 26B, and 26C in the OCV sequence — omitting volumes 21 and 27, which contain a scholarly introduction and appendices respectively\. For QE, seven volumes were likewise used \(38–43, excluding volume 37, which contains the scholarly introduction\)\.

Both texts were available in TEI\-XML format; preprocessing involved extracting the main text and stripping all markup tags\. The analytical indexes were available as structured files recording, for each entry, its hierarchical level and the list of volumes and pages to which it applies, including footnotes\. QE contains four hierarchical levels, with 419index1, 1,237index2, 216index3, and 4index4entries; EM contains three levels, with 556index1, 1,972index2, and 230index3entries\. Individual entries may in turn reference multiple page ranges \(e\.g\.\`\`Acre, iii\.334, 355, 356, 366, 368, 377, 382n, 390; conquise par Saladin, iii\.326–29; prise d', iii\.335–36, 337n''\), so the total number of indexable label instances exceeds the number of entries\. This study focuses throughout on level 1 entries unless otherwise specified\. Cross\-references \(entries preceded byvoir\) and entries referring exclusively to footnotes were excluded, the latter because our analysis operates at the level of the page rather than the note\.

The indexes were converted from their standard format \(e\.g\.\`\`abeilles, ii\.39–46, v\.370, 380, 450, vii\.449, 475''\) into a label format suitable for machine learning training \(e\.g\.\`\`abeilles,2,39; abeilles,2,40;…"\), producing datasets in which each page of text is associated with the set of index labels assigned to it by the professional indexer — a multi\-label classification task in any standard sense\.

The datasets were divided into training, validation, and test subsets in an approximate 80/10/10 ratio, using an iterative stratification method designed for multi\-label data\[szymanski2017network,sechidis2011stratification\]as implemented in the Scikit\-multilearn library\[szymanski2019scikit\], in order to achieve the best possible distributional balance across subsets\. For QE, this yielded 2,279/365/368 training/validation/test samples across 419 labels; for EM, 1,684/258/241 samples across 866 labels\.

A defining characteristic of both datasets — and a central challenge for model training — is the extreme skew in label frequency\. In QE, 75% of labels appear fewer than 7 times across the entire corpus; in EM, 75% appear fewer than 23 times\. At the page level, the distributions are similarly sparse: in both datasets, approximately 37% of pages carry a single label, and around 80% carry 3 or fewer\. Figures[1](https://arxiv.org/html/2607.09316#S3.F1)and[2](https://arxiv.org/html/2607.09316#S3.F2)illustrate these distributions\. This degree of label sparsity poses a substantial challenge for supervised learning: for the majority of index terms, the model has access to only a handful of positive training examples from which to learn reliable classification criteria\.

![Refer to caption](https://arxiv.org/html/2607.09316v1/distr_labels_EM.png)

![Refer to caption](https://arxiv.org/html/2607.09316v1/hist_labels_EM.png)

Fig\. 1:Label frequencies for the EM dataset\.![Refer to caption](https://arxiv.org/html/2607.09316v1/distr_labels_QE.png)

![Refer to caption](https://arxiv.org/html/2607.09316v1/hist_labels_QE.png)

Fig\. 2:Label frequencies for QE the dataset\.
## 4Methodology and Results

The experimental methodology is organised into three successive phases, each designed to narrow the field of candidate approaches before committing to full\-scale training\. In the first two phases, we compare models and architectures of increasing size to identify the most promising configuration; in the third, we conduct final training runs with the selected model and evaluate its capacity to generalise across corpora\. Throughout, we report the results of additional experiments involving modifications to training procedure, dataset composition, and model configuration, in order to characterise the sensitivity of results to these choices\.

### 4\.1Comparison of Medium\-Sized Models

The first phase of comparison was conducted on the EM dataset using models of approximately 5 billion parameters\. Two broad architectural approaches were explored in sequence: encoder\-based models with classification heads, and generative decoder models producing labels directly as text output\.

For the encoder\-based approach, pre\-trained LLMs were used as frozen feature extractors, with a series of fully connected linear layers added on top\. The final layer, activated by a softmax function, contained one neuron per label, yielding independent probability estimates for each class\. Four models were evaluated: Qwen3\-4B, Llama\-3\.2\-3B, Gemma\-3\-4B, and Mistral\-7B\-v0\.3, using the following Hugging Face checkpoints respectively:Qwen/Qwen3\-Embedding\-4B,meta\-llama/Llama\-3\.2\-3B,google/gemma\-3\-4b\-pt, andmistralai/Mistral\-7B\-v0\.3\. The classification head consisted of stacked Linear–ReLU–Dropout blocks, with the first layer sized proportionally to the encoder output by a scale factor hyperparameter, and subsequent layers reducing in size by a factor of2\(i\+1\)2^\{\(i\+1\)\}, whereiidenotes the layer index\. Encoder weights were frozen throughout; only the classification head was trained\. The loss function was weighted binary cross\-entropy, with per\-class weights computed as the ratio of negative to positive counts, to compensate for label imbalance\. For training, we used an AdamW optimiser with a LinearLR scheduler decaying the learning rate by a factor of 0\.1 over the course of training\. All models were implemented in PyTorch, using the Sentence Transformers333[https://sbert\.net](https://sbert.net/)and Transformers444[https://huggingface\.co/docs/transformers](https://huggingface.co/docs/transformers)libraries\. Hyperparameter tuning was conducted with Optuna,555[https://optuna\.org](https://optuna.org/)searching over dropout rate, learning rate, weight decay, batch size, first\-layer scale factor, and number of layers; models were trained for five epochs during tuning and up to 15 epochs thereafter\.

The results of this first comparison are shown in Table[1](https://arxiv.org/html/2607.09316#S4.T1)\. Performance across the encoder\-based models was modest, with F1 scores ranging from 0\.24 to 0\.34, reflecting the difficulty of the task and the limitations of a frozen encoder for a label space of this granularity and sparsity\.

These results motivated a shift to a generative decoding formulation, in which models produce the predicted label set directly as a structured text string rather than via a classification head\. This approach accommodates larger models more naturally and — as our subsequent results confirm — yields substantially better performance\. In this second approach, we compared an OpenAI proprietary model with several open\-source alternatives\.

The proprietary model used was GPT\-4\.1\-mini \(gpt\-4\.1\-mini\-2025\-04\-14\)\. Given the cost of fine\-tuning on texts of this length, experiments were limited to the mini variant\. OpenAI does not publish parameter counts, architectural details, or training data for its models; we assume GPT\-4\.1\-mini to be broadly comparable in scale to the five\-billion\-parameter models evaluated above, and include it here for reference accordingly\. Fine\-tuning used the OpenAI JSONL format\. The prompt provided in the user content field was as follows:\`\`You are an expert in French historical text classification\. You need to classify the following French text about Voltaire's historical works by assigning relevant thematic labels\. Text to classify:\{text\}\. Available labels:\{labels\}\. Instructions: 1\. Read the text carefully\. 2\. Identify all relevant themes present in the text\. 3\. Return only a Python list of label names that apply to this text\. 4\. Be precise — only include labels that are clearly relevant to the content\.''Structured JSON output was enforced via a response schema specifying apredicted\_labelsarray\. The model was trained for 3 epochs with automatic batch size and learning rate multiplier\.

The results for GPT\-4\.1\-mini are included in Table[1](https://arxiv.org/html/2607.09316#S4.T1)\. With an F1 of 0\.44, the generative formulation outperforms all encoder\-based models, confirming the value of the architectural shift\.

Several modifications to the GPT\-4\.1\-mini fine\-tuning procedure were explored, none of which produced meaningful improvements\. Expanding the context window to include the preceding and following pages alongside the target page yielded an F1 of 0\.44\. Testing different numbers of training epochs from one to five produced a marginal improvement at five epochs \(F1 = 0\.46\)\. A label partitioning experiment, in which the full label set was divided into four subsets of approximately 216 labels each and four independent models trained accordingly, increased recall to 0\.55 but caused precision and F1 to fall to 0\.21 and 0\.30 respectively — a result consistent with the hypothesis that smaller label sets reduce precision by increasing false positive rates across the combined output\. These negative results are informative in themselves: they suggest that the limiting factor at this scale is not context size, training duration, or label set granularity, but model capacity\.

Tab\. 1:Evaluation results of training of medium\-sized machine learning models on the EM dataset\.
### 4\.2Comparison of Large Quantised Models

The second phase scaled up to models of approximately 15 billion parameters, evaluated on the QE dataset\. To manage computational cost, all open\-source models were used in 4\-bit quantised form\. GPT\-4\.1\-mini was retained for cross\-phase comparison\. All models were used as generative text decoders with the same prompt as in Section[4\.1](https://arxiv.org/html/2607.09316#S4.SS1)\.

The models compared were: GPT\-4\.1\-mini, Qwen3\-14B, Gemma\-3\-12B, Phi\-4\-14B, Mistral\-Small\-3\.2\-24B, and Mistral\-Large\-123B, using the following Hugging Face checkpoints:unsloth/Qwen3\-14B\-unsloth\-bnb\-4bit,unsloth/gemma\-3\-12b\-it\-unsloth\-bnb\-4bit,unsloth/phi\-4\-unsloth\-bnb\-4bit,unsloth/Mistral\-Small\-3\.2\-24B\-Instruct\-2506\-unsloth\-bnb\-4bit, andunsloth/Mistral\-Large\-Instruct\-2407\-bnb\-4bit\. Open\-source models were fine\-tuned using Low\-Rank Adaptation \(LoRA\)\[hu2022lora\]with rankr=16r=16andα=16\\alpha=16, without dropout, for three epochs\. Training used the TRL library's666[https://huggingface\.co/docs/trl](https://huggingface.co/docs/trl)SFT Trainer with Unsloth framework integration,777[https://huggingface\.co/docs/trl/unsloth\_integration](https://huggingface.co/docs/trl/unsloth_integration)a learning rate of2×10−42\\times 10^\{\-4\}, weight decay of 0\.001, a linear scheduler, and AdamW8bit as optimiser\.

Results are shown in Table[2](https://arxiv.org/html/2607.09316#S4.T2)\. The Mistral family achieved the strongest performance, with Mistral\-Large\-123B reaching an F1 of 0\.65 and Mistral\-Small\-3\.2\-24B following closely at 0\.64\. Given the negligible performance gap and the substantial difference in computational requirements, Mistral\-Small\-3\.2\-24B was selected for the final phase\.

Tab\. 2:Evaluation results of training of large\-sized machine learning models on the QE dataset\.As in the previous phase, a series of methodological variants were explored using the QE dataset\. Combining GPT\-4\.1\-mini and Mistral\-Small\-3\.2\-24B sequentially — using the former's output labels as additional input to the latter for confirmation or rejection — produced no meaningful improvement \(F1: 0\.5525 to 0\.5558\), suggesting that the sequential architecture adds latency without complementary benefit\. Extending Mistral\-Small\-3\.2\-24B training to fifteen epochs produced only marginal gains \(F1: 0\.6431 to 0\.6488\), indicating that the model converges rapidly\. Converting all text to lowercase prior to training, motivated by the hypothesis that capitalised proper nouns were exerting undue influence on label assignment, likewise produced no significant change \(F1 = 0\.6479\)\. Finally, augmenting the dataset withindex2andindex3entries — renamed by combining each term with its parentindex1headword to avoid expanding the label space — produced virtually no improvement \(F1: 0\.6431 to 0\.6457\)\. Taken together, these results suggest that performance at this scale is relatively robust to the modifications explored, and that further gains are more likely to come from architectural choices or label space restructuring than from training procedure adjustments\.

### 4\.3Final Training and Out\-of\-Distribution Evaluation

The final phase used Mistral\-Small\-3\.2\-24B in its 4\-bit quantised configuration for two experiments: an assessment of the effect of label frequency filtering, and a cross\-corpus out\-of\-distribution evaluation\.

For the label filtering experiment, low\-frequency labels were removed from both datasets prior to training: in QE, labels appearing fewer than 5 times were excluded, reducing the label set from 419 to 257 \(a reduction of 39%\); in EM, labels appearing fewer than three times were excluded, reducing the label set from 866 to 423 \(51%\)\. Despite removing a large proportion of labels, the total number of indexed page instances was affected only modestly — by 5% for QE and 10% for EM — since low\-frequency labels account for a small share of overall label assignments\. Filtering produced consistent improvements: F1 rose from 0\.64 to 0\.67 for QE, and from 0\.42 to 0\.47 for EM\. While the gains are not dramatic, they confirm that the presence of labels with very few training examples introduces noise that measurably degrades overall performance\.

For the out\-of\-distribution experiment, models trained on one corpus were evaluated on the other: the QE\-trained model was applied to EM, and the EM\-trained model to QE\. The results, shown in Table[3](https://arxiv.org/html/2607.09316#S4.T3), reveal a consistent performance drop of approximately 10–20 percentage points in F1 when moving from in\-distribution to out\-of\-distribution evaluation\. This is to be expected given that the two corpora have entirely distinct label sets — the model must not only transfer its general classification behaviour but map between two different controlled vocabularies\. That any degree of cross\-corpus generalisation is observed at all is nonetheless encouraging, and suggests that the models are learning something about the structure of humanistic indexing beyond the specific labels of a single work\.

Tab\. 3:In\- and Out\-of\-distribution evaluation results for Mistral\-Small\-3\.2\-24B, 4\-bit quantised version\.

## 5Discussion

The quantitative results presented above warrant closer examination before their implications are drawn out, since the evaluation framework systematically understates model performance in ways that are worth making explicit\. A predicted label was counted as false whenever it diverged from the human\-attributed label derived from the print index — regardless of its semantic appropriateness\. This means that a model prediction can be both plausible and penalised simultaneously\. One such example is a page from the QE article\`\`Abeilles''in which Voltaire considers the social and economic organisation of a bee colony\. The human indexer assigned the labels\`\`abeilles''and\`\`anthropomorphisme''; the model adds\`\`science expérimentale'', which is entirely pertinent, since Voltaire discusses the close observations of his contemporary Jean\-Baptiste Simon over a period of twenty years\[voltaire\_ocv\_38, p\. 41\]\. Similarly, a page of the article\`\`Superstition''defends Anthony Ashley Cooper, third Earl of Shaftesbury, and his grandfather the first Earl, from recent literary attacks, contrasting their quiet lifestyle with the assassination of kings carried out or attempted by religious fanatics\. The human indexer attributed four labels to this page\[voltaire\_ocv\_43, p\. 319\]:\`\`philosophie'',\`\`raison humaine'',\`\`superstition'', and\`\`régicide''; the model produces the first two, but substitutes\`\`société civile''for\`\`régicide''\. It could reasonably be argued that regicide is a subcategory of civil society — and while the substitution loses a degree of semantic specificity,\`\`société civile''is far from wrong\. It must always be borne in mind that thematic indexing, however expertly conducted, is inherently subjective: no two professional indexers will produce precisely the same labels or structure for a text of any length or complexity\.

In places, the model produces results that compare favourably with the print index\. In chapter 81 of EM, which addresses customs and lifestyle in thirteenth\- and fourteenth\-century Europe, the human indexer attributes only the term\`\`courage''to a short page that nevertheless contains a thematically rich conclusion, comparing the relative poverty of France in the period with the riches of the Italian mercantile city\-states, where convenience and opulence reigned\. These advantages, Voltaire adds, gave rise to genius — as they do to courage\[voltaire\_ocv\_24, p\. 267\]\. The model has identified\`\`courage''but more pertinently also includes\`\`Italie''and\`\`luxe''\(luxury\)\. That France, mentioned more peripherally than Italy, does not feature as a label is particularly encouraging\.

The effect of page length on model performance is worth examining\. The page just discussed comes at the end of a print chapter, with blank space beneath the final lines of text; it contains only fifty\-nine words across 7 short lines\. Other pages carry relatively little primary text owing to the space taken up by lengthy editorial notes and variant readings \(see Figure[3](https://arxiv.org/html/2607.09316#S5.F3)\)\. A shorter text might be expected to produce less reliable labelling\. When results are sorted by word count, however, this hypothesis is not consistently borne out\. Among the twenty shortest EM pages in the results, four receive labels that match those of the human indexer exactly, and a further nine at least partially overlap or constitute plausible alternatives\. In a few cases the presence of a chapter title at the head of the page — which would have been ignored by the indexer — accounts for certain model\-predicted labels: the opening of EM chapter 5, entitled\`\`De la Perse, au temps de Mahomet le prophète, et de l'ancienne religion de Zoroastre'', predictably generates the label\`\`zoroastrisme''even where the body text on that page contains no reference to Zoroastrianism\[voltaire\_ocv\_22, p\. 97\]\. There are also instances where the brevity of a page genuinely defeats the model: at the end of EM chapter 103, a page of twenty\-four words —\`\`tout aux autres; les servant par avarice, les détestant par fanatisme, se faisant de l'usure un devoir sacré\. Et ce sont nos pères\!''\[voltaire\_ocv\_25, p\. 185\]— forms the tail end of a longer development on the Jewish nation, which the human indexer has placed under\`\`Juifs''\. Without the preceding context, the model not unreasonably assigns\`\`avarice''and\`\`sentiments haine''\.

![Refer to caption](https://arxiv.org/html/2607.09316v1/OCV_vol24_page193_crop2.png)Fig\. 3:Example of a page of EM containing a small amount of primary text\.Alongside results that compare positively with the print index, or are at least equal in validity, while being different, there are inevitably several that are less satisfactory\. In the QE article\`\`Supplices''\(Torture\), Voltaire illustrates his argument with examples of historical figures subjected to torture following dubious accusations of poisoning\[voltaire\_ocv\_43, p\. 332\]\. The print index gives the label\`\`empoisonnement''\(poisoning\), which the model also supplies, while adding\`\`gloire''\(glory\) and\`\`imposture''\(deception\)\. These last two terms clearly have their origins in single words that are peripheral to the main subject: at one point Voltaire addresses those who supported the accusations of poisoning but named different accomplices:\`\`Accordez\-vous donc, pauvres imposteurs''\(Get your stories straight, miserable deceivers\), he exclaims\. In the next paragraph, Voltaire refers in passing to the accusers being jealous of the glory of Holy Roman Emperor Charles V\. Neither word provides justifiable grounds for the labels assigned\.

A particularly clear instance of the model being misled by surface features occurs in the QE article\`\`Alexandre'', on Alexander the Great, where Voltaire adds a dig at Alexander's questions to the Gymnosophists in India, which he describes as worthy of theMercure galant, the late seventeenth\- and early eighteenth\-century literary gazette that eventually became theMercure de France\. The page as a whole\[voltaire\_ocv\_38, p\. 186\]concerns ancient history and the reliability of Plutarch's testimony, yet the model assigns it the label\`\`galanterie''— evidently on the basis of the periodical title\. To test whether capitalised proper nouns and titles were exerting undue influence on label assignment more generally, the full dataset was converted to lowercase and the experiment rerun, as noted in Section[4\.2](https://arxiv.org/html/2607.09316#S4.SS2), but this produced no significant change in performance\.

These two cases — one involving direct address to an imagined interlocutor, the other a humorous historical comparison — begin to take us beyond the domain of straightforward thematic content into the territory of Voltaire's rhetoric\. Voltaire is a literary writer, master stylist, and polemicist; rhetorical effects permeate his prose even when he is writing history or essays, where his voice and perspective were key selling points for contemporary readers\[menant2021, p\. 62\]\. This is particularly true of QE, a compendium of articles written late in life on a wide variety of topics, many of them serious but some mischievous or playful\. Like its better\-known precursor theDictionnaire philosophique\(DP\), some articles are written in dialogue form\. In the case of\`\`Liberté''and\`\`Liberté de penser'', the themes emerge clearly enough despite the dialogic format\[voltaire\_ocv\_42B, p\. 47; 53\]\. Irony is a more intractable challenge, as are allegory, periphrasis, and comparison, all of which can lead the model away from the underlying topic\. The QE article\`\`Gargantua''\[voltaire\_ocv\_42A, pp\. 1–5\]would have offered a particularly instructive test case: a masterclass in oblique polemic, it mounts an indirect attack on the Old Testament and biblical apologetics under the guise of a ludicrous discussion of the existence of the giant hero of Rabelais's novel \(for an analysis of this article, see\[sareil1983, pp\. 351–56\]; see also Pink's introductory note in\[voltaire\_ocv\_42A, p\. 1\]\)\. Unfortunately it does not feature in our test split\. The DP article\`\`Guerre''does, however, and it offers a comparable example\. The article pillories priests and the Church for celebrating bloody military victories while ordinarily preaching against vice — a line of attack that, in the eighteenth century, required careful circumlocution\. The word\`\`prêtre''\(priest\) does not appear, nor is the Bible named; Voltaire identifies his subjects through periphrasis and comic detail\. His priests are referred to as\`\`harangeurs'', their vestments and speech described as if observed by a visitor from an alien culture\. The model, thoroughly bamboozled by Voltaire's verbal acrobatics, produces the labels\`\`démonisme''and\`\`nourriture''\(food\), having fixed on two items in an accumulation of things the priest condemns in peacetime:\`\`que Polyeucte et Athalie sont les ouvrages du démon; qu'un homme qui fait servir sur sa table pour deux cents écus de marée un jour de carême, fait immanquablement son salut''\(thatPolyeucteandAthalie\[plays by Corneille and Racine\] are the work of the devil; that a man guarantees his salvation by having 200 crowns worth of seafood \[instead of meat, is implied\] served at his table on a day in Lent\)\[voltaire\_ocv\_36, p\. 191\]\. The labels\`\`guerre'',\`\`Église catholique'', and\`\`religion''were available in the label set but were not applied\. It is worth noting that\`\`prêtres''was not available, since in the human\-generated index it appears as a sub\-entry under\`\`religion''— a reminder that the constraint of working with level 1 labels limits the model's expressive range in ways that are not always visible in the quantitative results\.

The treatment of quotations presents a related but distinct difficulty\. Some QE articles contain long extracts from verse\. In some cases the model copes reasonably well: a section of the article\`\`Athéisme''reproduces almost in full Voltaire's 1769 poemÉpître à l'auteur du livre des Trois Imposteurs, a polemical work that tackles its subject directly and without heavy recourse to figurative language; the model's proposed labels closely match those of the human indexer\[voltaire\_ocv\_39, p\. 174\]\. The poetry present in the text is not always Voltaire's own, however\. He peppers his articles with prose and verse quotations from writers ranging from classical antiquity to his own time\. The QE article\`\`Langues''contains a passage from Molière'sL'Amour médecin, included to illustrate the historical usage of the French word\`\`rate''rather than the more recent English borrowing\`\`spleen''\[voltaire\_ocv\_42B, pp\. 7–8\]; the human indexer classifies page 8 as comparative linguistics, while the model opts for\`\`corps humain''and\`\`rire''\(human body and laughter\)\. Similarly, quotations from Virgil and Seneca — for which Voltaire helpfully provides French translations — skew the interpretation of an article on the development of the notion of hell \(\`\`Enfer''\), where the presence of the word\`\`imposture''in Voltaire's translation generates a corresponding but non\-pertinent label from the model\[voltaire\_ocv\_41, p\. 109\]\. A related problem emerges in the article\`\`Bouffon, burlesque'', where Voltaire quotes passages from Scarron's playsDon Japhet d'ArménieandLe Virgile travesti: what the human indexer places under\`\`goût''\(taste\) and\`\`style''is assigned the label\`\`populace''by the model \(OCV 39:448\)\. This same label recurs in a page of the article\`\`Esprit''\(spirit, wit\), which the human indexer has again categorised as\`\`style''; the model adds\`\`populace''and\`\`traduction''\(translation\) — the latter because Voltaire mentions the term\`\`traduction''in passing, merely to observe that only bad Latin translations of Hebrew writings use the wordspiritusin the sense of an ethereal being\[voltaire\_ocv\_41, p\. 244\]\. The question of how to handle quotations is not straightforwardly resolved\. On the one hand, quotations are tagged as such in the TEI\-XML files underlying our experiments, and it would be technically feasible either to mark them as a feature or to exclude them from the text passed to the model\. On the other hand, quotations do not always serve merely illustrative purposes: Voltaire frequently includes them for their content, in which case they are thematically relevant and ought to be taken into account\. The distinction is usually clear to a human indexer but remains difficult to encode\.

When the models were applied to corpora for which no index previously existed — QE labels to the DP and theLettres philosophiques, EM labels to theHistoire de Charles XII— performance declined appreciably, though in the absence of pre\-existing human\-generated labels any evaluation was necessarily subjective\. The choice of source label set matters considerably: applying EM labels to theHistoire de Charles XIIproduced assignments that skewed heavily towards broad geographical headwords \(\`\`Russie'',\`\`Pologne'',\`\`Suède'',\`\`Estonie'',\`\`Finlande''\), which are of limited utility to a reader seeking to navigate the narrative and analysis of Charles XII's campaigns\. As has long been recognised in indexing theory, one of the indexer's primary tasks is the selection of headwords that collectively capture the distinctive thematic profile of the work in question\[duncan2022, pp\. 259–60\]; a label set designed for one work will almost inevitably prove ill\-fitting for another, even when the two are generically related\. The cross\-corpus experiments described in Section[4\.3](https://arxiv.org/html/2607.09316#S4.SS3)confirm this at the quantitative level: the 10–20 percentage point drop in F1 under out\-of\-distribution conditions reflects not only the change in text but the mismatch between label sets\.

The DP article\`\`Dieu''offers a useful illustration of the cumulative difficulties the model faces when working outside its training distribution\. Cast as a dialogue between Logomacos, a theologically sophisticated visitor from Constantinople, and Dondindac, a simple provincial believer from the south of France, the article interweaves fiction, philosophy, and polemic in ways that resist straightforward thematic assignment\[voltaire\_ocv\_36, pp\. 21–28\]\. As Logomacos interrogates Dondindac, the model produces a sequence of labels —\`\`athéisme''for a page about the practice of prayer;\`\`ignorance''for Dondindac's account of what he knows about God;\`\`philosophie''for a page in which Dondindac protests incomprehension before increasingly abstruse theological questions;\`\`déisme''where Dondindac does indeed profess deistic beliefs;\`\`anthropomorphisme''for a parable about a mole in dialogue with a mayfly; and\`\`athéisme''again for a page of 16 words that can hardly be said to have a discernible theme at all\. While\`\`déisme''is plausible, the others are not, and the repeated assignment of\`\`athéisme''to pages with little or no connection to atheism is difficult to account for\. In the article\`\`Destin'', for example, this label\`\`athéisme''is assigned to a page effectively limited to a cross\-reference:\`\`plutôt, si vous pouvez examiner paisiblement avec moi ce que c'est, passez à la lettre L\.''\[voltaire\_ocv\_36, p\. 19\]\.

It is worth stepping back from the qualitative texture of individual results to reflect on a more fundamental question of method\. Technological innovations have historically played a constitutive role in the genesis of indexing as a practice: it was only with the emergence of the printed book, with its multiple identically paginated copies, that producing an index became a worthwhile investment of labour and funds relative to the manuscript paradigm that preceded it\[abbamonte2023, p\. 466; 474\]\. The index, in turn, gave rise to a new mode of engagement with text — one in which readers could navigate selectively, following their own paths through a work according to a consultation model rather than reading in linear fashion from start to finish\[blair2010, pp\. 117–21\]\. In the experiments described here, we retained the page as the unit of analysis, both to maintain compatibility with the print indexes and to allow the human\-generated labels to serve as a control\. Yet from a semantic perspective, the page is an unsatisfactory unit of analysis: again and again, examining the results, we encountered pages containing only the beginning or end of a longer passage whose subject cannot be reliably identified without the surrounding context\. The codex, with its individually numbered pages, made indexing possible; but in a digital environment, there is no technical reason to be bound by this artefact\. Indexing at the level of the paragraph — or of the semantically coherent passage, as identified by the model itself — would be both more natural and more accurate\. This is perhaps the most significant methodological implication of our experiments, and one that points towards a genuinely different approach to digital indexing in future work\.

## 6Conclusions

The experiments reported here demonstrate both the promise and the current limits of machine learning as a tool for automatic thematic indexing of large\-scale literary and historical corpora\. Fine\-tuned generative LLMs — and in particular the 4\-bit quantised Mistral\-Small\-3\.2\-24B model — outperform encoder\-based architectures with classification heads across all evaluation conditions, achieving F1 scores of 0\.64 and 0\.43 on the QE and EM datasets respectively, rising to 0\.67 and 0\.47 when low\-frequency labels are filtered out\. These figures represent lower bounds on real performance: as the qualitative analysis makes clear, the automatic evaluation framework penalises semantically valid predictions that happen to differ from the human\-generated ground truth, and manual inspection consistently reveals a proportion of \`\`false'' generated labels that are in fact plausible or even superior alternatives to the print index labels\.

The qualitative analysis is, in many respects, the more revealing contribution of this study\. Quantitative metrics capture what the model gets wrong relative to a fixed standard; the close reading of individual results illuminateswhyit goes wrong and what that tells us about the nature of the task\. The findings point consistently towards a cluster of difficulties that are unlikely to be resolved by scaling alone: the heavy skew of label distributions towards rare terms, the semantic richness and generic diversity of the source texts, and above all the rhetorical complexity of Voltaire's prose — his irony, his misdirection, his habit of embedding serious argument in playful or fictional frames\. These are not incidental features of the corpus but constitutive ones, and they suggest that the gap between model performance and professional indexing practice reflects something deeper than a lack of training data or model capacity\. As\[duncan2022\]observes, ''every subject index worth its salt must, in an important sense, be a work of the imagination'' \(p\. 259\)\. As such, the index is in many ways the product of interpretative judgements that current models can approximate but not yet replicate\.

It is also worth reflecting on our choice of corpus\. Voltaire was a natural starting point for this work, given the existence of the new digital editions of the OCV, complete with some professionally indexed volumes and the desirability of extending reliable thematic access to the unindexed works within and beyond it\. He may not, however, be the easiest author with whom to begin\. His tendency to weave disparate topics together, to write allusively, to deploy irony as a primary argumentative instrument, and to mean something rather different from what he says — all qualities that were irresistible to his eighteenth\-century readers — make him a demanding subject for any system that relies on surface\-level semantic signals\. That the models perform as well as they do under these conditions is, on reflection, encouraging\. All the more reason, then, to want to open up more paths into his vast body of writing — but which is perhaps not the most forgiving place to begin developing a fledgling approach to automatic indexing\.

Perhaps the most significant methodological lesson of these experiments concerns not the models themselves but the unit of analysis\. As argued in the Discussion, the page — inherited from the print tradition and retained here for reasons of comparability — is a poor semantic unit for indexing purposes\. A move towards paragraph\- or passage\-level indexing, exploiting the structural markup already present in the TEI\-XML source files, would both improve model performance and produce indexes more naturally suited to digital navigation\. Combined with the extension of trained models to unindexed works — the original motivation for this project — such a shift would represent a meaningful step towards the kind of genuinely structured access to large literary corpora that full\-text search alone cannot provide\. The problem of making Voltaire's 15 million words navigable to the non\-specialist reader remains open; these experiments suggest that it is tractable, and indicate where the most productive directions for future work might lie\.

The authors would like to thank Digital Scholarship @ Oxford \(DiSc\) for funding and supporting this project, with the particular involvement of Andrew Cusworth \(DiSc and Bodleian Libraries, University of Oxford\)\. Access to the digitised files of Voltaire's text with page numbers and the printed index labels was provided by the Voltaire Foundation \(University of Oxford\)\. We would also like to thank Haris Zia, who contributed to the project in its early stages\.

This project has been partially funded by Digital Scholarship @ Oxford \(DiSc\)\.

The authors state no conflict of interest\.

Author agreement/declaration::This statement certifies that all authors have seen and approved the final version of the submitted manuscript\. This manuscript has not received prior publication and is not being considered for publication elsewhere\. Data availability statement:The software developed during this project, as well as the machine learning models we have trained, are publicly available as open\-source in the following repository:[https://github\.com/Digital\-Scholarship\-Oxford/digitally\-indexing\-voltaire](https://github.com/Digital-Scholarship-Oxford/digitally-indexing-voltaire)

## References