MIL-BERT: Classification of Arbitrarily Large Text with Performance and Explanatory Guarantees

# MIL-BERT: Classification of Arbitrarily Large Text with Performance and Explanatory Guarantees
Source: [https://arxiv.org/html/2608.20636](https://arxiv.org/html/2608.20636)
###### Abstract

Many text classification decisions are viable based on constituent excerpts alone\. Taking inspiration from the field of multiple instance learning, we present an algorithm for training a neural network to classify text by selecting such excerpts\. We show that our approach is also scalable with demonstrated learning against samples with nearly 1M tokens\. We evaluate our methods on 7 datasets with emphasis on long\-textual collections that far exceed the encoding limit of our base model\. We present state\-of\-the\-art results with this algorithm on 3 datasets: identification of political bias in news outlets, trigger warnings in long stories, and demographic characteristics of authors in tweet collections\. Furthermore, the model trained on weakly\-labeled collections of text \(bags\) generalizes to accurately classify constituent, smaller instances\. Besides a new state\-of\-the\-art for these problems, this approach is one of the few neural methods to excel in these datasets\.

## 1Introduction

In the current state of the art, the most accurate approaches for textual classification and regression \(C&R\) are based on language models in the transformer family[37](https://arxiv.org/html/2608.20636#bib.bib41)\. However, canonical transformers have fixed\-length positional embeddings and computation and memory usage that grows quadratically with sequence length\. Many practical applications of C&R using popular foundation models make compromises such as truncation when processing long documents, implicitly assuming that sufficient information for accurate prediction can be found early in a document\. On the other hand, transformer adaptations for long documents such as Infinite Transformer[21](https://arxiv.org/html/2608.20636#bib.bib15)are relatively expensive to apply and process the entire text, opaquely and possibly needlessly\. We surmise that many problems of textual C&R can be successfully addressed by instead identifying one or a few passages that support a particular prediction—a phenomenon we call*signal locality*\. For problems exhibiting signal locality, we can replace the wholesale processing of an overlarge text with a search for indicative excerpts of modest size\. This amounts to a*multiple instance learning*\(MIL\) framing of the problem\.

MIL is a paradigm of machine learning in which collections of*instances*called*bags*\. A bag is classified as positive as long as one of its instances is positive\. For example, MIL has been applied to the task of identifying cancerous image patches \(instances\) within larger, whole\-slide images based on a slide\-level \(bag\) label[28](https://arxiv.org/html/2608.20636#bib.bib2)\. If an image contains cancer, then there is at least one image patch that contains cancer\. If an image does not contain cancer, then no image patches contain cancer\. An example application of MIL to text involves identifying civil unrest based on classifying collections of tweets for particular days and countries[9](https://arxiv.org/html/2608.20636#bib.bib1)\. Inspired by these approaches, we develop an algorithm for learning to classify arbitrarily long text with performance and explainability guarantees\.

This paper makes the following contributions:

- •We describe an efficient approach for classifying long texts based on a MIL framing, involving a fixed set of “selectors” that identify the passages most relevant to a particular prediction\.
- •In experiments involving a range of previously proposed long\-text classification problems, we demonstrate that this excerptive approach, MIL\-BERT, is competitive with whole\-document alternatives, in several cases surpassing previously reported accuracies\.
- •We describe the advantages of MIL\-BERT for explainability, arising from the requirement that it identify key, easily reviewable passages that drive the classification decision\.

## 2Method

### 2\.1Preliminaries

In the canonical formulation of MIL[4](https://arxiv.org/html/2608.20636#bib.bib11), we perform binary classification of*bags*, each bagX=\{x1,…​xn\}X=\\\{x\_\{1\},\.\.\.x\_\{n\}\\\}containing a set of instancesxix\_\{i\}with labelsyiy\_\{i\}\. A bagXXreceives positive labelY=1Y=1if any of its constituent instancesxix\_\{i\}has labelyi=1y\_\{i\}=1, in effect making MIL a sparse learning problem\. During training, only bag labels are given, and the objective is to learn an effective instance classifier\. Subsequent variants of MIL have prioritized bag labels[12](https://arxiv.org/html/2608.20636#bib.bib3)or presented noisy instance labels at training time[28](https://arxiv.org/html/2608.20636#bib.bib2)\. We explore problems with both visible \(trigger warnings\) and implicit instance labels \(demographic profiling\)\.

### 2\.2Algorithm

We present a novel algorithm for textual C&R problems exhibiting signal locality, representing long documents as arbitrarily large bags of excerpts\. In this study, we segment a document into excerpts\(x1,x2,…\)\(x\_\{1\},x\_\{2\},\.\.\.\)with a rolling window and embed each excerpt with a RoBERTa model[19](https://arxiv.org/html/2608.20636#bib.bib26), using the embedding of the start symbol token \(<s\>\) from the final layer\. Our key innovation comes from the insight that, during training, intermediate activations of instances are kept in memory to later calculate the gradients even if they are not relevant to the gradients such as with a max pool shown in equation[1](https://arxiv.org/html/2608.20636#S2.E1)\.

∇max\(f\(x1,…\)\)=∇max\(f\(argmax\(f\(x1,…\)\)\)\)\\nabla max\(f\(x\_\{1\},\.\.\.\)\)=\\nabla max\(f\(argmax\(f\(x\_\{1\},\.\.\.\)\)\)\)\(1\)
We achieve this mathematical substitution through what we refer to as the*double\-pass pooling trick*\. In the first pass, without gradients, the model selects key excerpts with an argmax function we call aselector\. In the second pass, we run the model with gradients on the subset selected in the first pass\. This allows the model to calculate the gradient for an arbitrary number ofnnpoints withkkmemory; the tradeoff is that we useO⁡\(n\+k\)O\(n\+k\)compute\. We depict this process in Figure[1](https://arxiv.org/html/2608.20636#S2.F1)\.

![Refer to caption](https://arxiv.org/html/2608.20636v1/milber-figures-v3.jpg)Figure 1:Documents are broken up into excerpts which are encoded with a BERT\-style model\. In the first pass \(dotted line\), a selector layer finds extreme points without gradients, saving memory\. On a second run with gradients \(dark lines\), these are passed through the selector layer, a Gumbel\-softmax operation, and onto a classifier layer\.All selectors in this study are based on a linear layer of dimensionkk\. We perform an affine transformation of all excerpts in the bag\(W​X\+b\)\(WX\+b\)and select the most extreme excerpts in each dimension, similar to a maxpool operation\. There are multiple ways in which the excerpts thus selected can be combined to generate a prediction\. Here, we consider two methods in this study\. In theshared feature mapselector, parameters are shared between selection and classification, the output dimension of the affine transformation matches the number of classes and the top value for each dimension ultimately becomes the logit for that class\. With variants selecting the top\-k instances and aggregating scores[9](https://arxiv.org/html/2608.20636#bib.bib1), this classical MIL approach also learns an instance classifier from bag labels\. In thefeature treeselector, each dimension of the affine transformation resembles a feature of a decision tree with a threshold of 0; if index 0 is greater than 0, then we next consider index 1; otherwise, index 2\. This allows the use of1\+2k−11\+2^\{k\-1\}selection decisions with only k instances selected, reducing memory usage\. Geometrically, each selector index represents a hyperplane and the sign indicates which side of the hyperplane the bag is on, but this does not bear a direct relationship to classes as in the shared feature map selector\. Thefeature treemethod loses instance\-orientation to fit higher\-order patterns\. Other selectors are possible, but for double\-pass pooling to work, selector methods must be*idempotent*, i\.e\. application of selectors must yield the same result between the first and second passes\.

To enable a general range of classifier layers, we specify that selectors emit the embeddings of selected instances and pass them to the classifier\. This requires an argmax operation which is not differentiable during training\. Fortunately, the Gumbel\-softmax trick allows a differentiable selection of individual instances[13](https://arxiv.org/html/2608.20636#bib.bib4)by adding noise to the vector in order to create an approximately one\-hot vector for the maximum value in the vector\. This has been extended to perform top\-k selection in a differentiable manner among other discrete operations[26](https://arxiv.org/html/2608.20636#bib.bib5)\. This enables more powerful classification layers such as transformers or self\-attention to be applied over instances as has been done in MIL research[31](https://arxiv.org/html/2608.20636#bib.bib40)\. For classifiers layers in this study, we use a simple linear layer but its dimensionality varies\. The shared classifier layer shares all its parameters with its selector, using top\-k excerpts to score each class with k equal to one\. The feature tree classifier uses the values of selected indices as input into a linear layer \(1\+2k−1×c​l​a​s​s​e​s1\+2^\{k\-1\}\\times classes\) with the rest being zero\.

## 3Datasets

We explore the capability of our algorithms on MIL with identifiable instance labels, MIL with implicit labels, and long\-text documents\.

First, we investigate classic MIL applications where we train a classifier on bags of text and then evaluate its accuracy onbothbag and instance classification where labels are identifiable\. The task is to identify trigger warnings in stories[43](https://arxiv.org/html/2608.20636#bib.bib7), specifically the 2023 shared task[42](https://arxiv.org/html/2608.20636#bib.bib31)111https://pan\.webis\.de/clef23/pan23\-web/trigger\-detection\.html\. This dataset only has labels at the story \(bag\) level, but, in later work, they searched for passages \(instances\) within the stories that were likely to contain trigger words and labeled them with respect to 1 of 8 finegrained binary labels[40](https://arxiv.org/html/2608.20636#bib.bib8); we evaluate instance accuracy for the 7 labels that match between the two datasets\. The second MIL dataset comes from CLEF 2023 task 3222https://gitlab\.com/checkthat\_lab/clef2023\-checkthat\-lab/\-/tree/main/task3which concerns identifying political bias in news sources \(task B: bag\) and individual articles \(task A: instance\)[7](https://arxiv.org/html/2608.20636#bib.bib9)\. Both of these benchmarks have standard training, validation and test partition data\.

Second, we present results in demographic profiling of celebrity authors available from PAN2019@CLEF333https://pan\.webis\.de/clef19/pan19\-web/celebrity\-profiling\.htmlwhere gender, occupation, birthyear, and fame\-level must be predicted based on tweet collections[41](https://arxiv.org/html/2608.20636#bib.bib30)\. Demographic profiling matches the use\-case of MIL for learning over noisy data with implicit labels\. The phrase "my husband" is indicative of gender, but it is not definitive, so excerpts in this domain defy explicit labeling\. The dataset has a training and test partition, so we use 20% of the training partition for validation with stratified sampling\. For the birthyear task, we bucket years according to its task\-specific F1\-based scoring function that features wider buckets for older writers\. Using MSE loss, we normalize the range by subtracting 1920 and dividing by 100\. Anonymizing URL’s improved results in this dataset\.

As a final area of interest, we also consider the application of these methods to a long document classification benchmark[25](https://arxiv.org/html/2608.20636#bib.bib16)that contains results for previous algorithms: Hyperpartisan[15](https://arxiv.org/html/2608.20636#bib.bib10), 20Newsgroups, EURLEX\-57K[5](https://arxiv.org/html/2608.20636#bib.bib28), and Book Text[2](https://arxiv.org/html/2608.20636#bib.bib29)\. Train, validation and test partitions have been specified for each dataset444https://github\.com/amazon\-science/efficient\-longdoc\-classification\. Whereas the previous datasets have appeared as shared tasks, this long document benchmark features 5 repeated runs, so we have done so as well\.

We describe the number of classes for each dataset in Table[1](https://arxiv.org/html/2608.20636#S4.T1)and the lengths of their documents in Table[2](https://arxiv.org/html/2608.20636#S4.T2)\.

## 4Hyperparameters

Table[3](https://arxiv.org/html/2608.20636#S5.T3)lists the hyperameters set for each experiment, primarily chosen by heuristics as follows\. We have the size of the excerpt windows W, a stride S, the choice of selector \(shared or feature tree\) which has a corresponding classification layer\. We mostly used the shared classifier since it generally performs better and requires less tuning \(n​\_​e​x​c​e​r​p​t​s=c​l​a​s​s​e​sn\\\_excerpts=classes\)\. For simplicity, we targeted a window size of 256 tokens with a stride at14\\frac\{1\}\{4\}the window size for variation recalling strides in CNN’s with shifting augmentation akin to translation augmentation in image processing[32](https://arxiv.org/html/2608.20636#bib.bib32)\. The so\-calledshared feature mapselector classifiers must have a number of excerpts directly proportional to the number of classes which can lead to dramatic increases in memory usage, mandating shorter excerpts for extreme classification problems as in the case of EURLEX with over 4000 classes\. For CLEF23, we use a shared selector with a top\-k value of 3 since it was trained on only 817 examples \(news sources\) where there are more biased instances \(articles\) to learn from each bag \(source\)\. The other exception for W isPAN19\-Birthyear, which required the*feature tree*selector because the*shared selector*underperformed during the hyperparameter tuning runs\. The variance in losses and reweighting is explained by the key performance metrics; for single\-class datasets emphasizing macro\-f1, we used class\-based reweighting, but not for those emphasizing accuracy or micro\-f1 \(eg\. longtext benchmark and CLEF23B\)\. For multilabel datasets, we used ZLPR loss that balances multi\-label loss with respect to class frequency[33](https://arxiv.org/html/2608.20636#bib.bib25)\. We trained models with model checkpointing for the key metric per dataset in tables \(eg\. accuracy, micro\-f1, macro\-f1\)\. We used learning rate annealing\. For the Gumbel\-softmax, tau was set to 1 with the soft sampling technique and a decay rate of12\\frac\{1\}\{2\}over 48 epochs\. We used RoBERTa large as the model to embed excerpts[19](https://arxiv.org/html/2608.20636#bib.bib26)\.

Table 1:The datasets used in these experiments which include longer documents and collections of text\. All include bag labels, while some also include instance labels\.Table 2:The length of the datasets in RoBERTa tokens
## 5Results

We now describe the results of our experiments, starting with classification performance and memory and compute usage\. We follow with a visual analysis of how selector chosen instances and subsequent bag\-level representations contrast with the other instances in a bag\. We conclude with an analysis of the the explanatory qualities of instances chosen by the selectors\.

### 5\.1Classification Results

Table 3:Hyperparameters for each experimentFirst, we share results in MIL problems where a model trained on bag\-level information achieves state\-of\-the\-art results on that task but also generalizes to instance passage classification\. We achieve state\-of\-the\-art classification results for news source \(CLEF 3B\) which generalizes to accurately classify individual articles though it lags behind the state\-of\-the\-art methods trained on the more plentiful, article\-labeled data; see Table[4](https://arxiv.org/html/2608.20636#S5.T4)\. In Table[5](https://arxiv.org/html/2608.20636#S5.T5), we share results for trigger\-warning detection at the story\-level \(bag\) as well as passage\. We achieve state\-of\-the\-art results at the bag\-level and respectable accuracy at the passage\-level\. We present results on the PAN 2019 author profiling task where our solution creates new state\-of\-the\-art results for gender and occupation but not age or fame in Table[8](https://arxiv.org/html/2608.20636#S5.T8)\. We report performance of our approach on long\-document classification in Table[8](https://arxiv.org/html/2608.20636#S5.T8)\. We present an ablation study on 20News with 5 seeded runs for insight into hyperparameters in Table[3](https://arxiv.org/html/2608.20636#S5.T3)\. For this dataset, we can see that window size of 256 performs better than 128, and the shared classifier performs best; feature tree does best with 6 excerpts, indicating potential overfitting with 4 more\.

We present the memory and compute time profile of our approach compared with some popular approaches in Figure[2](https://arxiv.org/html/2608.20636#S5.F2), including LongFormer[3](https://arxiv.org/html/2608.20636#bib.bib14)\. For each approach, we calculated the maximum resident VRAM during backpropagation on a single instance, clearing cached memory between each inference\. Results conform with our description ofO⁡\(k\)O\(k\)memory usage withO⁡\(n\+k\)O\(n\+k\)compute; in direct comparison, a naive max pooling MIL\-BERT approach without the double pass pooling trick consumes 6 times the memory to consider only 48 contexts of 256 tokens compared to 8\-class MIL\-BERT\. As for compute, we can see continual growth for longer documents for MIL\-BERT while other approaches stop using more once they reach their truncation limit\. Training on a single instance can approach 100 seconds for 1M tokens\. Surprisingly, we observe that the fully trained 8\-class MIL\-BERT PAN19 occupation model uses less memory than the equivalent 8\-excerpt, naive MIL\-BERT, but single\-passages may be selected multiple times by a selector, reducing memory requirements below the max in the second pass\. Compared to other long\-text approaches, MIL\-BERT covers far more tokens than even long\-context encoders such Longformer\-large and DeBERTa base with much less memory usage\.

### 5\.2Chosen Instances Visual Analysis

We now present a visual analysis to contrast how selector chosen instances contrast with the other instances in a bag, starting with the shared classifier\. We present a UMAP projection[22](https://arxiv.org/html/2608.20636#bib.bib44)of the 3\-dimensional logits for each instance \(excerpts\) from 15 sources scored with the shared classifier from the CLEF23 3B test set in Figure[3](https://arxiv.org/html/2608.20636#S5.F3)[11](https://arxiv.org/html/2608.20636#bib.bib34)\. To illustrate the local bag structure, we select 2 neighbors, a minimum distance of 0\.05 and Euclidean distance as our metric\. In addition to selected and unselected excerpts, we present the bag representation which is the average of 3 points in each direction to create the logits for each bag\. As would be expected, selected points line on the convex hull of each bag\. Given they contribute the largest feature score magnitudes, they would be the most descriptive for the classification boundary\. Instances of same source tend to be tightly grouped together–likely due to topical and ideological consistencies across documents as well as duplicate content between excerpts from striding\.

### 5\.3Explainability

By construction the feature selector picks instances with the largest magnitude along each feature dimension\. In the classifiers used for this work, these instances are the ones that drive the final decision as they account for the largest classification score contribution across the feature dimensions\. Thus instances picked by the feature selector guaranteed to be the ones that would be most impactful on the classifier’s decision\. Similarly, during learning they are also the ones that contribute most strongly to the weights update\.

Given this, selector scores for excerpts provide qualitative insights about the decision function\. In addition, they can be analyzed quantitatively with correlation or conventional explainability techniques\.

First, we can analyze in aggregate by considering how their individual scores in each dimension \(Figure[4](https://arxiv.org/html/2608.20636#S5.F4)\) correlate with the classes of concern\. We present selector dimensions of individual selector scores with separate histograms for each decade of birthyear\. We calculate a moderate correlation between these selector scores and birthyear \(0\.42\-0\.62\)\. We notice that all excerpts are beyond plane 0 \(positive\) while plane 5 has scores above and below zero\. This plane bifurcates authors loosely around 1970 and passes them on to planes 12 and 13, representing specialization like CNN feature maps\. Second, individual excerpts and selector scores can be interrogated for explainability purposes with layer integrated gradients[35](https://arxiv.org/html/2608.20636#bib.bib35)as provided by Captum[16](https://arxiv.org/html/2608.20636#bib.bib36)\. In Figure[5](https://arxiv.org/html/2608.20636#S5.F5), we present a visualization of the selected left\-bias excerpt from a liberal news source by a shared selector\. Gradients were calculated against the word embeddings to calculate how they impact the output of the selector, not the overall classification decision\. Putting these methods together, it is possible to characterize how individual passages contribute to the final score and why they are selected\.

Table 4:Results for the 2023 political bias dataset\.DatasetBagInstanceMacro\-F1Micro\-F1Average AccOurs0\.370\.750\.709[30](https://arxiv.org/html/2608.20636#bib.bib12)0\.3520\.74—5 fold multiclass——0\.699Mixtral8x7B——0\.747Table 5:The performance of classifiers trained on bag\-level labels along with performance on instance\-labeled datasets for PAN23 trigger warning dataset\. Instance\-level labels are majority vote decisions for 7 classes from the in\-domain subset: violence, death, abduction, racism, homophobia, misogyny, ableism\.Table 6:The performance of classifiers for long document benchmark with its partitions with previous results[25](https://arxiv.org/html/2608.20636#bib.bib16)\. Metrics averaged over 5 runs with different seeds\. Our results include standard deviation, unlike previously reported benchmark results\.Table 7:Results from an ablation study on 20News dataset for 5 seeds![Refer to caption](https://arxiv.org/html/2608.20636v1/memory-usage.png)

![Refer to caption](https://arxiv.org/html/2608.20636v1/cpu-usage.png)

Figure 2:Memory usage of algorithms compared with truncation on the PAN 2019 dataset with batch size 1 and no optimizer\. Truncation limits appear in all other approaches\. Time to perform backpropagation calculated as the number of tokens varies\.![Refer to caption](https://arxiv.org/html/2608.20636v1/bag-visualization-bias-n-2-d-0.05-vsmall.png)

![Refer to caption](https://arxiv.org/html/2608.20636v1/bag-visualization-source-n-2-d-0.05-vsmall.png)

Figure 3:UMAP projection of collections of excerpts from various sources in the CLEF23 3B dataset with few neighbors to preserve local structure\. The top 3 instances are selected per dimension and average to create the bag score\. Selected instances are on the convex hull in classification boundary space of 3 dimensions\.![Refer to caption](https://arxiv.org/html/2608.20636v1/selector-visualization-age.png)Figure 4:Visualization of selector scores from the age profiling feature tree model\. Each histogram is a selector hyperplane and the X\-axis is the selector score\. Individual selectors specialize to features relevant in age profiling\.![Refer to caption](https://arxiv.org/html/2608.20636v1/milbert-viz-liberal-v2.png)Figure 5:An excerpt classified as obvious left bias from a left\-bias source with tokenwise attribution under layer integrated gradients\. Blue indicates positive attribution toward the selector score, rather than the classification decision\. White is near neutral and red contributes against the selector\. This excerpt is heavily influenced by "fossil fuel industry\."Macro\-F1Dataset:GenderAgeFameOccupOurs0\.7830\.5710\.430\.549[29](https://arxiv.org/html/2608.20636#bib.bib27)0\.7260\.6180\.5510\.515Table 8:Performance on the PAN 2019 celebrity profiling dataset\. Age is scored as a modified classification task with a hit\-window ranging from 2\-6 years\.

## 6Discussion

We have demonstrated that MIL\-BERT is an efficient and explainable algorithm for long texts or C&R problems with a natural MIL structure, delivering competitive accuracy\. We treat identification of trigger warnings, political bias and demographic characteristics as MIL problems for the first time to achieve new state\-of\-the\-art results, and documenting the accuracy of instance classifiers trained only via bag labels where possible \(trigger warnings and political bias detection\)\. While setting a new state\-of\-the\-art in these tasks, we present competitive results in the long document benchmark, where no single method is dominant\. Finally, we demonstrate ways selector scores of excerpts can be interpreted individually and in aggregate\.

We bring progress to three datasets that have proven unwieldy for conventional neural text classification techniques due to memory limits\. In trigger warning detection, previous state\-of\-the\-art methods either use a frozen RoBERTa model as a feature extractor[30](https://arxiv.org/html/2608.20636#bib.bib12)or train a RoBERTa model on just the first and last 512 tokens[34](https://arxiv.org/html/2608.20636#bib.bib37)\. In a previous version of this task, an XGBoost[6](https://arxiv.org/html/2608.20636#bib.bib42)baseline outperforms Longformer because truncation leads to missing trigger warnings[43](https://arxiv.org/html/2608.20636#bib.bib7)\. Likewise, traditional feature engineering approaches led the top team to success in author profiling, with negative results for neural approaches from 3 out of six teams[41](https://arxiv.org/html/2608.20636#bib.bib30)\. Finally, we present improved results for source bias classification CLEF23 3B task[7](https://arxiv.org/html/2608.20636#bib.bib9)where the previous best team[1](https://arxiv.org/html/2608.20636#bib.bib38)used CatBoost[27](https://arxiv.org/html/2608.20636#bib.bib39)\. We have shown progress in terms of task metrics but also in terms of finding a suitable neural model\.

Our success is limited on some of the tasks\. In the long\-document classification benchmark, our approach is best in only one dataset, but no existing approach is dominant\. Based on author profiling results, we conclude that MIL\-BERT is most suitable for MIL tasks and is otherwise merely competitive in long\-text C&R problems\. Though we have improved performance in predicting an author’s gender or occupation, our approach still lags behind in predicting birthyear and fame, presumably because these tasks do not exhibit the signal locality for which MIL\-BERT is designed\.\. Gender and occupation can be identified by excerpts, which borders on the task of self\-disclosure detection[36](https://arxiv.org/html/2608.20636#bib.bib6)\. Another limitation of this study is the exploration of hyperparameters\. We have tuned hyperparameters for the birthyear dataset, but further tuning could still improve results\. If we had normalized birthyear to match the hit\-window function used in evaluation, we would likely improve performance by weighting discrepancies for younger authors more heavily\. One reason for this under\-exploration is that we cannot escape the fundamental limits to our algorithm\. Computation time inevitably increases with document length and the number of excerpts increases the memory burden\.

We leave it to future work to improve selector and classifier layers\. For example, we have experimented with auto\-regressive selection with transformers in other work\. At each timestep, the input is the previously selected instance and the output is the means to select the next instance\. For now we recursively compute key\-instances, but future work could detach gradients as the points are sampled forO⁡\(n\)O\(n\)compute, though not for all selection methods such as feature tree\. As for classification layers, we have used transformers in other problems with more training data since this can overfit\. There are obvious dangers of this algorithm for mining protected characteristics of individuals, but if offers new C&R applications across multiple documents such as entity typing[8](https://arxiv.org/html/2608.20636#bib.bib43)and MIL\-based, distant information extraction[39](https://arxiv.org/html/2608.20636#bib.bib33)\. The double\-pass pooling technique can be applied to many problems besides C&R, especially those involving retrieval and matching tasks, which we also leave to future work\. Towards facilitating replication and future work, we plan to release our library containing MIL\-BERT\.

## 7Related Work

Taking inspiration from approaches in MIL and explainable models, we present a solution which offers scaling to long\-document classification with explainability guarantees\.

### 7\.1Multiple\-instance Learning

MIL has been extensively applied in biology, computer vision, and natural language processing[4](https://arxiv.org/html/2608.20636#bib.bib11)on problems involving collections of labeled instances \(molecular orientations, image subregions, or textual excerpts\)\. There are approaches which emphasize accuracy on either the bag or the instance such as[28](https://arxiv.org/html/2608.20636#bib.bib2)\. MIL with the accurate identification of key instances has long been applied to NLP problems such as event detection[38](https://arxiv.org/html/2608.20636#bib.bib19)\. There are true approaches to learning neural decision trees \(e\.g\.[14](https://arxiv.org/html/2608.20636#bib.bib24)\) and they have been extended to MIL problems[17](https://arxiv.org/html/2608.20636#bib.bib23), but they use all instances and self\-attention to produce an aggregate score\. PEDL applies a type of MIL to learn protein\-protein interactions based on mentioned contexts, but the authors could sample at most 100 contexts, ostensibly due to memory limitations[39](https://arxiv.org/html/2608.20636#bib.bib33)\. Likewise, other authors train separate instance and bag models[9](https://arxiv.org/html/2608.20636#bib.bib1)\. Our approach enables the processing of arbitrarily large bags under fixed memory usage\.

### 7\.2Explainability

There are many broad approaches to explainability in NLP: selector\-predictor architectures \(as here\), post\-hoc input perturbation, analysis of attention, and methods which analyze gradients[20](https://arxiv.org/html/2608.20636#bib.bib21)\. Before the Gumbel\-softmax became popular, the REINFORCE algorithm was used to overcome the inherent issues of calculating gradients over discrete selection to enable selector\-predictor models[18](https://arxiv.org/html/2608.20636#bib.bib22)\. The Gumbel\-softmax trick has been used to select top\-k phrases to use in calculating aspect\-oriented sentiment analysis[26](https://arxiv.org/html/2608.20636#bib.bib5)\. This technique has also been used to align textual phrases with image segments[44](https://arxiv.org/html/2608.20636#bib.bib20)\. We view this work as continued exploration of the established explainability of MIL approaches to text classification such as event detection \(eg\.[38](https://arxiv.org/html/2608.20636#bib.bib19)\), but we also identify new performance guarantees in the selector\-predictor paradigm with idempotent selectors\.

### 7\.3Long\-document classification

In long document classification, most research has pursued increasing the length of transformers attention mechanism such as Longformer[3](https://arxiv.org/html/2608.20636#bib.bib14)or the Infinite Memory transformer[21](https://arxiv.org/html/2608.20636#bib.bib15)\. By contrast, some work has pursued identifying key\-excerpts\. CogLTX resembles this work in that they find a memory efficient way to sample subsections of a document, but it uses two BERT models to do so: a judge which recalls selectors and a final reasoner model[10](https://arxiv.org/html/2608.20636#bib.bib13)\. Without predefined excerpts, the judge determines tokens sequences, concatenates them and passes them into a second model\. Furthermore, the unsupervised training method involves ablating subsections and calculating loss in order to rank relevant sections as compared to our geometric formulation with direct ties to the primary loss function\. Similarly, the BERT\+TextRank[25](https://arxiv.org/html/2608.20636#bib.bib16)baseline uses an unsupervised algorithm TextRank[23](https://arxiv.org/html/2608.20636#bib.bib17)to select relevant content to learn against\. ToBERT uses state\-of\-the\-art hierarchical document processing model with a transformer of BERT subsegments[24](https://arxiv.org/html/2608.20636#bib.bib18), but such approaches can be prohibitively expensive in terms of memory as in the trigger warning task[30](https://arxiv.org/html/2608.20636#bib.bib12)\.

## 8Conclusion

We have demonstrated an algorithm for learning to classify arbitrarily large collections of text which exchanges compute for memory savings with the double pass pooling trick and idempotent selectors\. Along the way, we present new state\-of\-the\-art results in trigger warning detection and political bias detection of sources, which can then be applied to classify constituent elements\. We present limited improvements to authorship profiling and competitive results in long document classification\. In the near\-term, we would like to see these techniques applied to more MIL problems, even those outside of text\. We have outlined some possible improvements to selectors and classifiers which we would like to apply to other problems with more data or other related problems such as retrieval\.

## 9Limitations

The instance labeled data for the MIL experiments regarding article bias and triggering passages are limited\. For the sake of full comparison with other approaches, we have not identified whether the bags containing these instances are seen in training\. Nevertheless, we believe this is reasonable since the bag\-labeled training data does not label instances explicitly\.

In\-depth profiling of memory usage of algorithms was impeded by the tendency of pytorch to reserve blocks of memory between samples even with calls to release cached memory\. We ran each model in its own process with a GPU and emptied cached CUDA memory: deleting items, running garbage collection, and resetting pytorch’s statistics\. However, memory reservation may make our numbers closer to the high\-water mark of GPU memory usage\. Also, it seems to be less accurate at the lower end of memory utilization\. Memory and compute requirements do increase with more classes due to more selected excerpts, but we do not observe the same expected multiple when considering the 3\-class MIL\-BERT classifier where it uses marginally less memory than the 8\-class classifier\. We believe this is due to limited visibility into memory allocation and fragmentation\.

## 10Ethics Statement

The capability of the algorithm to learn against arbitrarily large data presented here creates opportunities for performing classification problems that are otherwise infeasible\. By virtue of selecting passages, it is interpretable\. However, improved performance on author profiling indicates it could be used to infer protected characteristics of individuals if presented with such training data\. At the same time, it presents capabilities identifying trigger warnings\.

## 11Acknowledgments

This material is based upon work supported by the Intelligence Advanced Research Projects Activity \(IARPA\) under Contract \#2022\-22072200004 and the National Science Foundation \(NSF\) under Contract \#49100422C0013\. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author\(s\) and do not necessarily reflect the views of the IARPA or NSF\.

## References

- Azizovet al\.\(2023\)D\. Azizov, P\. Nakov, and S\. LiangFrank at checkthat\!\-2023: detecting the political bias of news articles and news media\.\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Bamman and Smith \(2013\)D\. Bamman and N\. A\. SmithNew alignment methods for discriminative book summarization\.arXiv preprint arXiv:1305\.1319\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p4.1)\.
- Beltagyet al\.\(2020\)I\. Beltagy, M\. E\. Peters, and A\. CohanLongformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[§5\.1](https://arxiv.org/html/2608.20636#S5.SS1.p2.1),[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- Carbonneauet al\.\(2018\)M\. Carbonneau, V\. Cheplygina, E\. Granger, and G\. GagnonMultiple instance learning: a survey of problem characteristics and applications\.Pattern Recognition77,pp\. 329–353\.Cited by:[§2\.1](https://arxiv.org/html/2608.20636#S2.SS1.p1.1),[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1)\.
- Chalkidiset al\.\(2019\)I\. Chalkidis, E\. Fergadiotis, P\. Malakasiotis, and I\. AndroutsopoulosLarge\-scale multi\-label text classification on EU legislation\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 6314–6322\.External Links:[Link](https://aclanthology.org/P19-1636/),[Document](https://dx.doi.org/10.18653/v1/P19-1636)Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p4.1)\.
- Chen and Guestrin \(2016\)T\. Chen and C\. GuestrinXgboost: a scalable tree boosting system\.InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining,pp\. 785–794\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Da San Martinoet al\.\(2023\)G\. Da San Martino, F\. Alam, M\. Hasanain, R\. N\. Nandi, D\. Azizov, and P\. NakovOverview of the clef\-2023 checkthat\! lab task 3 on political bias of news articles and news media\.\.InCLEF \(Working Notes\),pp\. 250–259\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p2.1),[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Del Corroet al\.\(2015\)L\. Del Corro, A\. Abujabal, R\. Gemulla, and G\. WeikumFinet: context\-aware fine\-grained named entity typing\.InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,pp\. 868–878\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p4.1)\.
- DeLuciaet al\.\(2023\)A\. DeLucia, M\. Dredze, and A\. L\. BuczakA multi\-instance learning approach to civil unrest event detection on twitter\.InProceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio\-political Events from Text,pp\. 18–33\.Cited by:[§1](https://arxiv.org/html/2608.20636#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.20636#S2.SS2.p4.1),[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1)\.
- Dinget al\.\(2020\)M\. Ding, C\. Zhou, H\. Yang, and J\. TangCogltx: applying bert to long texts\.Advances in Neural Information Processing Systems33,pp\. 12792–12804\.Cited by:[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- Healy and McInnes \(2024\)J\. Healy and L\. McInnesUniform manifold approximation and projection\.Nature Reviews Methods Primers4\(1\),pp\. 82\.Cited by:[§5\.2](https://arxiv.org/html/2608.20636#S5.SS2.p1.1)\.
- Ilseet al\.\(2018\)M\. Ilse, J\. Tomczak, and M\. WellingAttention\-based deep multiple instance learning\.InInternational conference on machine learning,pp\. 2127–2136\.Cited by:[§2\.1](https://arxiv.org/html/2608.20636#S2.SS1.p1.1)\.
- Janget al\.\(2016\)E\. Jang, S\. Gu, and B\. PooleCategorical reparameterization with gumbel\-softmax\.arXiv preprint arXiv:1611\.01144\.Cited by:[§2\.2](https://arxiv.org/html/2608.20636#S2.SS2.p5.1)\.
- Karthikeyanet al\.\(2021\)A\. Karthikeyan, N\. Jain, N\. Natarajan, and P\. JainLearning accurate decision trees with bandit feedback via quantized gradient descent\.arXiv preprint arXiv:2102\.07567\.Cited by:[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1)\.
- Kieselet al\.\(2019\)J\. Kiesel, M\. Mestre, R\. Shukla, E\. Vincent, P\. Adineh, D\. Corney, B\. Stein, and M\. PotthastSemEval\-2019 task 4: hyperpartisan news detection\.InProceedings of the 13th International Workshop on Semantic Evaluation,pp\. 829–839\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p4.1)\.
- Kokhlikyanet al\.\(2020\)N\. Kokhlikyan, V\. Miglani, M\. Martin, E\. Wang, B\. Alsallakh, J\. Reynolds, A\. Melnikov, N\. Kliushkina, C\. Araya, S\. Yan,et al\.Captum: a unified and generic model interpretability library for pytorch\.arXiv preprint arXiv:2009\.07896\.Cited by:[§5\.3](https://arxiv.org/html/2608.20636#S5.SS3.p3.1)\.
- Konstantinovet al\.\(2023\)A\. Konstantinov, L\. Utkin, and V\. MuliukhaMultiple instance learning with trainable soft decision tree ensembles\.Algorithms16\(8\),pp\. 358\.Cited by:[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1)\.
- Leiet al\.\(2016\)T\. Lei, R\. Barzilay, and T\. JaakkolaRationalizing neural predictions\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,J\. Su, K\. Duh, and X\. Carreras \(Eds\.\),Austin, Texas,pp\. 107–117\.External Links:[Link](https://aclanthology.org/D16-1011/),[Document](https://dx.doi.org/10.18653/v1/D16-1011)Cited by:[§7\.2](https://arxiv.org/html/2608.20636#S7.SS2.p1.1)\.
- Liu \(2019\)Y\. LiuRoberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692364\.Cited by:[§2\.2](https://arxiv.org/html/2608.20636#S2.SS2.p1.1),[§4](https://arxiv.org/html/2608.20636#S4.p1.1)\.
- Luoet al\.\(2024\)S\. Luo, H\. Ivison, S\. C\. Han, and J\. PoonLocal interpretations for explainable natural language processing: a survey\.ACM Computing Surveys56\(9\),pp\. 1–36\.Cited by:[§7\.2](https://arxiv.org/html/2608.20636#S7.SS2.p1.1)\.
- Martinset al\.\(2022\)P\. H\. Martins, Z\. Marinho, and A\. F\. Martins∞\\infty\-former: infinite memory transformer\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5468–5485\.Cited by:[§1](https://arxiv.org/html/2608.20636#S1.p1.1),[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- McInneset al\.\(2018\)L\. McInnes, J\. Healy, and J\. MelvilleUmap: uniform manifold approximation and projection for dimension reduction\.arXiv preprint arXiv:1802\.03426\.Cited by:[§5\.2](https://arxiv.org/html/2608.20636#S5.SS2.p1.1)\.
- Mihalcea and Tarau \(2004\)R\. Mihalcea and P\. TarauTextRank: bringing order into text\.InProceedings of the 2004 Conference on Empirical Methods in Natural Language Processing,D\. Lin and D\. Wu \(Eds\.\),Barcelona, Spain,pp\. 404–411\.External Links:[Link](https://aclanthology.org/W04-3252/)Cited by:[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- Pappagariet al\.\(2019\)R\. Pappagari, P\. Zelasko, J\. Villalba, Y\. Carmiel, and N\. DehakHierarchical transformers for long document classification\.In2019 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),Vol\.,pp\. 838–844\.External Links:[Document](https://dx.doi.org/10.1109/ASRU46091.2019.9003958)Cited by:[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- Parket al\.\(2022\)H\. Park, Y\. Vyas, and K\. ShahEfficient classification of long documents using transformers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 702–709\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p4.1),[Table 6](https://arxiv.org/html/2608.20636#S5.T6),[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- Pauluset al\.\(2020\)M\. Paulus, D\. Choi, D\. Tarlow, A\. Krause, and C\. J\. MaddisonGradient estimation with stochastic softmax tricks\.Advances in neural information processing systems33,pp\. 5691–5704\.Cited by:[§2\.2](https://arxiv.org/html/2608.20636#S2.SS2.p5.1),[§7\.2](https://arxiv.org/html/2608.20636#S7.SS2.p1.1)\.
- Prokhorenkovaet al\.\(2018\)L\. Prokhorenkova, G\. Gusev, A\. Vorobev, A\. V\. Dorogush, and A\. GulinCatBoost: unbiased boosting with categorical features\.Advances in neural information processing systems31\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Quet al\.\(2024\)L\. Qu, Y\. Ma, X\. Luo, Q\. Guo, M\. Wang, and Z\. SongRethinking multiple instance learning for whole slide image classification: a good instance classifier is all you need\.IEEE Transactions on Circuits and Systems for Video Technology\.Cited by:[§1](https://arxiv.org/html/2608.20636#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.20636#S2.SS1.p1.1),[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1)\.
- \[29\]V\. Radivchev, A\. Nikolov, and A\. LambovaCelebrity profiling using tf\-idf, logistic regression, and svm\.\.Cited by:[Table 8](https://arxiv.org/html/2608.20636#S5.T8.2.4.1)\.
- Sahinet al\.\(2023\)U\. Sahin, I\. E\. Kucukkaya, and C\. ToramanARC\-nlp at pan 2023: hierarchical long text classification for trigger detection\.Cited by:[Table 5](https://arxiv.org/html/2608.20636#S5.T5.2.4.1),[§6](https://arxiv.org/html/2608.20636#S6.p2.1),[§7\.3](https://arxiv.org/html/2608.20636#S7.SS3.p1.1)\.
- Shaoet al\.\(2021\)Z\. Shao, H\. Bian, Y\. Chen, Y\. Wang, J\. Zhang, X\. Ji,et al\.Transmil: transformer based correlated multiple instance learning for whole slide image classification\.Advances in neural information processing systems34,pp\. 2136–2147\.Cited by:[§2\.2](https://arxiv.org/html/2608.20636#S2.SS2.p5.1)\.
- Shorten and Khoshgoftaar \(2019\)C\. Shorten and T\. M\. KhoshgoftaarA survey on image data augmentation for deep learning\.Journal of big data6\(1\),pp\. 1–48\.Cited by:[§4](https://arxiv.org/html/2608.20636#S4.p1.1)\.
- Suet al\.\(2022\)J\. Su, M\. Zhu, A\. Murtadha, S\. Pan, B\. Wen, and Y\. LiuZlpr: a novel loss for multi\-label classification\.arXiv preprint arXiv:2208\.02955\.Cited by:[§4](https://arxiv.org/html/2608.20636#S4.p1.1)\.
- Suet al\.\(2023\)Y\. Su, Y\. Han, and H\. QiSiamese networks in trigger detection task\.\.InCLEF \(Working Notes\),pp\. 2782–2786\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Sundararajanet al\.\(2017\)M\. Sundararajan, A\. Taly, and Q\. YanAxiomatic attribution for deep networks\.InInternational conference on machine learning,pp\. 3319–3328\.Cited by:[§5\.3](https://arxiv.org/html/2608.20636#S5.SS3.p3.1)\.
- Umaret al\.\(2019\)P\. Umar, A\. Squicciarini, and S\. RajtmajerDetection and analysis of self\-disclosure in online news commentaries\.InThe World Wide Web Conference,pp\. 3272–3278\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p3.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2608.20636#S1.p1.1)\.
- Wanget al\.\(2016\)W\. Wang, Y\. Ning, H\. Rangwala, and N\. RamakrishnanA multiple instance learning framework for identifying key sentences and detecting events\.InProceedings of the 25th ACM International on Conference on Information and Knowledge Management,pp\. 509–518\.Cited by:[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1),[§7\.2](https://arxiv.org/html/2608.20636#S7.SS2.p1.1)\.
- Weberet al\.\(2020\)L\. Weber, K\. Thobe, O\. A\. M\. Lozano, J\. Wolf, and U\. LeserPEDL: extracting protein\-protein associations using deep language models and distant supervision\.Bioinformatics \(Oxford, England\)36\(Suppl\_1\),pp\. i490–i498\.Cited by:[§6](https://arxiv.org/html/2608.20636#S6.p4.1),[§7\.1](https://arxiv.org/html/2608.20636#S7.SS1.p1.1)\.
- Wiegmannet al\.\(2024\)M\. Wiegmann, J\. Rakete, M\. Wolska, B\. Stein, and M\. PotthastIf there’s a trigger warning, then where’s the trigger? investigating trigger warnings at the passage level\.arXiv preprint arXiv:2404\.09615\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p2.1)\.
- Wiegmannet al\.\(2019\)M\. Wiegmann, B\. Stein, and M\. PotthastOverview of the celebrity profiling task at pan 2019\.\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p3.1),[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Wiegmannet al\.\(2023a\)M\. Wiegmann, M\. Wolska, M\. Potthast, and B\. SteinOverview of the trigger detection task at pan 2023\.\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p2.1)\.
- Wiegmannet al\.\(2023b\)M\. Wiegmann, M\. Wolska, C\. Schröder, O\. Borchardt, B\. Stein, and M\. PotthastTrigger warning assignment as a multi\-label document classification problem\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12113–12134\.Cited by:[§3](https://arxiv.org/html/2608.20636#S3.p2.1),[§6](https://arxiv.org/html/2608.20636#S6.p2.1)\.
- Zhenget al\.\(2020\)K\. Zheng, W\. Liu, J\. Liu, Z\. Zha, and T\. MeiHierarchical gumbel attention network for text\-based person search\.InProceedings of the 28th ACM international conference on multimedia,pp\. 3441–3449\.Cited by:[§7\.2](https://arxiv.org/html/2608.20636#S7.SS2.p1.1)\.

## Appendix AAppendix

### A\.1Training details

We trained our models on a variety of machines\. For our longest text dataset, we used 6 A6000s for 3 days\. For the evaluation of how much memory is required and training time of a single bag we used a single RTX 3090\.