Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure

# Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure
Source: [https://arxiv.org/html/2607.11994](https://arxiv.org/html/2607.11994)
Deepika Giri2Avishek Kadel3Rabin Kumar Karki4Akbar Siami Namin1 Department of Computer Science1Texas Tech University1 Cumberland University2Yeshiva University3University of Cumberlands4 \{bipin\.chhetri, akbar\.namin2\}@ttu\.edudgiri25@students\.cumberland\.edu akadel@mail\.yu\.edurkarki34351@ucumberlands\.edu

###### Abstract

Classifying cybersecurity vulnerabilities using the Common Weakness Enumeration \(CWE\) taxonomy is challenging due to extreme class imbalance and strong hierarchical dependencies among weakness categories\. Although oversampling techniques such as Synthetic Minority Oversampling Technique \(SMOTE\) and Adaptive Synthetic Sampling \(ADASYN\) are widely adopted to mitigate class imbalance, their effectiveness for hierarchical CWE text classification remains largely unexplored\. This paper proposes a Hierarchy\-Aware RoBERTa framework that explicitly incorporates CWE structural information through learnable parent\-class embeddings, preserving taxonomic consistency\. Our experiments demonstrate that synthetic interpolation in high\-dimensional embedding spaces violates the inherent parent\-child constraints of the CWE hierarchy, offering only marginal benefits for classical ML models while consistently degrading deep learning architectures\. Evaluated on a CWE Research Concept dataset, the proposed model achieves a weighted F1\-score of0\.760\.76without data augmentation, outperforming all baselines with notable gains on minority classes, including theClasscategory whose F1\-score improves from0\.490\.49to0\.600\.60over the BERT baseline\. Our results suggest that hierarchy\-aware representation learning is a more principled alternative to oversampling for structured vulnerability classification\.

## IIntroduction

Mitigating weaknesses has become increasingly necessary as software systems grow in complexity and adversaries adopt sophisticated attack techniques\. Software and hardware weaknesses cataloged in the CWE\[[15](https://arxiv.org/html/2607.11994#bib.bib1)\]enable systematic vulnerability assessment, mitigation planning, and security decision making\[[4](https://arxiv.org/html/2607.11994#bib.bib52)\]\. The CWE taxonomy organizes weaknesses into abstraction levels such asBase,Class, andPillar, where upper\-level nodes capture broad concepts, and lower\-level nodes represent specific details\. In practice, a small number of weakness types dominate the corpus, while many rare but high\-impact categories remain significantly underrepresented\.

Transformer\-based encoders such as BERT\[[8](https://arxiv.org/html/2607.11994#bib.bib66)\]and RoBERTa\[[14](https://arxiv.org/html/2607.11994#bib.bib65)\]have become strong baselines for CWE text classification, capturing bidirectional contextual dependencies that outperform earlier Convolutional Neural Network \(CNN\)\[[17](https://arxiv.org/html/2607.11994#bib.bib67)\]and Recurrent Neural Network \(RNN\) architectures\[[5](https://arxiv.org/html/2607.11994#bib.bib68)\]\. However, class imbalance remains a fundamental challenge that degrades performance on rare CWE categories, and CWE labels carry explicit parent\-child dependencies\[[11](https://arxiv.org/html/2607.11994#bib.bib70)\]\. For instance, aVariantis always a child of aBase, meaning misclassification at higher abstraction levels cascades downward to finer\-grained classes\.

Research on hierarchical classification emphasizes that maintaining parent\-child relationships is crucial for building reliable prediction models\[[11](https://arxiv.org/html/2607.11994#bib.bib70)\]\. Traditional approaches, such as random oversampling and undersampling\[[18](https://arxiv.org/html/2607.11994#bib.bib58)\], rebalance label distributions but risk overfitting on minority samples or losing valuable information from the majority class\. More advanced methods, SMOTE\[[3](https://arxiv.org/html/2607.11994#bib.bib24)\]and ADASYN\[[9](https://arxiv.org/html/2607.11994#bib.bib25)\], generate synthetic minority samples in feature space\. SMOTE interpolates between minority instances to reduce classifier bias toward majority labels, while ADASYN concentrates synthesis on harder boundary regions where misclassifications are likely\. The specific contributions of our work are as follows:

1. 1\.We empirically evaluate ML \(RF, SVM\), deep learning \(CNN, BiGRU\), and transformer\-based \(BERT\) models under SMOTE and ADASYN resampling on an imbalanced hierarchical CWE dataset\.
2. 2\.We demonstrate that synthetic interpolation in high\-dimensional embedding spaces violates CWE parent\-child constraints, marginally helping classical ML models while consistently degrading deep learning models\.
3. 3\.We introduce a Hierarchy\-Aware RoBERTa model that injects CWE structural priors through learnable parent\-class embeddings for end\-to\-end joint modeling of semantic content and hierarchical structure\.
4. 4\.Our proposed model achieves a weighted F1\-score of0\.760\.76without augmentation, outperforming all baselines withClassF1 improving from0\.490\.49to0\.600\.60over BERT\.

This paper is structured into the following sections\. Section[II](https://arxiv.org/html/2607.11994#S2)presents the related work\. Section[III](https://arxiv.org/html/2607.11994#S3)presents the technical background of the models\. Sections[IV](https://arxiv.org/html/2607.11994#S4)and[V](https://arxiv.org/html/2607.11994#S5)contain the methodology and experimental setup, respectively\. Section[VI](https://arxiv.org/html/2607.11994#S6)presents the results\. Section[VII](https://arxiv.org/html/2607.11994#S7)presents the discussion and limitations\. Section[VIII](https://arxiv.org/html/2607.11994#S8)concludes the paper with future work\.

## IIRelated Work

### II\-ACWE\-Based Vulnerability Classification

Prior work primarily treats CWE classification as a supervised multiclass problem in data\-rich settings\. Chhetri et al\.\[[4](https://arxiv.org/html/2607.11994#bib.bib52)\]showed that BERT outperformed CNN, LSTM, and HAN architectures in predicting the consequences of cyber attacks from the descriptions of CWE, but performance degraded under sparse and imbalanced conditions\. VulnBERTa\[[16](https://arxiv.org/html/2607.11994#bib.bib8)\], a hierarchical RoBERTa\-based classifier trained on reports from the National Vulnerability Database \(NVD\), achieved strong accuracy on frequent CWE classes but significant degradation on rare labels\. Contreras et al\.\[[6](https://arxiv.org/html/2607.11994#bib.bib5)\]evaluated BI\-LSTM and BiGRU models on the Software Assurance Reference Dataset \(SARD\) and the NVD with strong overall results but struggled to distinguish semantically overlapping CWE classes, highlighting the persistent barrier of long\-tail label distributions\.

### II\-BSemantic and Hierarchical Classification

Previous work\[[13](https://arxiv.org/html/2607.11994#bib.bib11)\]applied BERT\-based cross\-encoders with binary chaining to the CWE View\-1003 hierarchy, improving accuracy on high\-cardinality label sets but remaining limited in tail classes\. V2W\-BERT\[[7](https://arxiv.org/html/2607.11994#bib.bib12)\]demonstrated that label embeddings and textual CWE definitions improve representations for single\-label CWE assignment\. Classical ML approaches including Random Forests \(RF\)\[[2](https://arxiv.org/html/2607.11994#bib.bib63)\]and Support Vector Machines \(SVM\)\[[10](https://arxiv.org/html/2607.11994#bib.bib64)\]perform well on frequent weaknesses but consistently degrade on rare ones, motivating approaches that explicitly model label structure\.

### II\-CClass Imbalance in Datasets

SMOTE\[[3](https://arxiv.org/html/2607.11994#bib.bib24)\]and ADASYN\[[9](https://arxiv.org/html/2607.11994#bib.bib25)\]have proven effective for classifiers based on classical ML and CNN, and recent work has incorporated oversampling directly into deep learning training loops\[[12](https://arxiv.org/html/2607.11994#bib.bib32)\]\. However, the behavior of SMOTE and ADASYN on hierarchically structured CWE text embeddings, where parent\-child constraints impose structural dependencies that linear interpolation cannot respect, remains largely unexplored, a gap this paper directly addresses\.

## IIITechnical Background

### III\-AMachine Learning Models

#### III\-A1Random Forest

\(RF\)\[[2](https://arxiv.org/html/2607.11994#bib.bib63)\]constructs an ensemble of decision trees on bootstrapped subsets of training data, aggregating predictions to reduce variance and overfitting\. Each tree recursively selects splits that maximize impurity reduction, performing well on structured feature spaces, but struggling on high\-dimensional sparse text representations\. Here, the best split is chosen as the feature and threshold that leads to the largest decrease in impurity between the parent and its children\.

#### III\-A2Support Vector Machine

\(SVM\)\[[10](https://arxiv.org/html/2607.11994#bib.bib64)\]learns a maximum\-margin hyperplane that separates classes in a high\-dimensional feature space\. A linear kernel is well suited for sparse text representations, making SVM a strong baseline for short\-text classification tasks such as CWE descriptions\.

### III\-BDeep Learning Models

#### III\-B1Convolutional Neural Networks

\(CNN\)\[[17](https://arxiv.org/html/2607.11994#bib.bib67)\]applies 1\-D convolutional filters over token embeddings to capture local n\-gram patterns, followed by max\-pooling to retain the most salient activations\. While effective for short range dependencies, CNNs are sensitive to perturbations in embedding space introduced by synthetic oversampling\.

#### III\-B2Bidirectional Gated Recurrent Unit

\(BiGRU\)\[[5](https://arxiv.org/html/2607.11994#bib.bib68)\]processes sequences in both forward and backward directions, concatenating hidden states at each position to capture full contextual dependencies\. The gated mechanism controls information flow across time steps, making BiGRU effective for modeling order\-sensitive vulnerability descriptions\.

### III\-CTransformers Based Models

#### III\-C1Bidirectional Encoder Representations from Transformers

\(BERT\)\[[8](https://arxiv.org/html/2607.11994#bib.bib66)\]uses stacked encoder layers with multi\-head self\-attention, pretrained via masked language modeling\. Robustly Optimized Bidirectional Encoder Representations from Transformers \(RoBERTa\)\[[14](https://arxiv.org/html/2607.11994#bib.bib65)\]improves on BERT by removing next sentence prediction, adopting dynamic masking, and training on substantially larger corpora, yielding stronger contextual representations across NLP benchmarks\. SecureBERT\[[1](https://arxiv.org/html/2607.11994#bib.bib62)\]extends RoBERTa through continuous pretraining on a large cybersecurity corpus and serves as the encoder backbone for our proposed model\.

## IVMethodology

### IV\-AOversampling

The dataset exhibited severe class imbalance across five classes, i\.e\.,Base\(393\),Variant\(219\),Class\(83\),Compound\(8\), andPillar\(5\)\. Oversampling was applied exclusively to the training set to prevent data leakage\. SMOTE equalized all classes to 393 samples usingkk\-neighbors=2=2, while ADASYN produced near parity counts \(Compound395,Base393,Pillar392,Class387,Variant362\) by concentrating synthesis on harder boundary regions\. Figure[1](https://arxiv.org/html/2607.11994#S4.F1)illustrates the resulting class distributions before and after resampling\.

![Refer to caption](https://arxiv.org/html/2607.11994v1/Figure/oversampling.png)Figure 1:Class distribution before and after SMOTE and ADASYN oversampling\.
### IV\-BModel Architectures

![Refer to caption](https://arxiv.org/html/2607.11994v1/Figure/hb_framework.png)Figure 2:Overview of the Hierarchy\-Aware RoBERTa framework\.For classical baselines, RF was set with 100 decision trees \(nn\-estimators=100=100\) and a fixed random seed \(random\-state=42=42\)\. The SVM classifier was configured to use a linear kernel, which is appropriate in high\-dimensional, sparse feature representations\.

For deep learning, CNN used 128 convolutional filters of kernel sizes \[4,6,6\] withReLUactivation and adaptive max pooling, while BiGRU used a hidden dimension of 128 with stacked recurrent layers to capture long\-range contextual dependencies\. Both models mapped tokens to 256\-dimensional embeddings trained end\-to\-end, fusing a 16\-dimensional categorical status embedding via a 128\-dimensional fusion layer \(ReLU, dropout 0\.2\)\. BERT\-base \(uncased\) used the\[CLS\]pooled output fused with a 16\-dimensional status embedding, followed by a 128\-dimensional linear layer \(ReLU, dropout 0\.2\)\. All models were trained with a batch size of 8 for 15 epochs\.

### IV\-CHierarchy\-Aware RoBERTa

We propose a Hierarchy\-Aware RoBERTa architecture \(Algorithm[1](https://arxiv.org/html/2607.11994#alg1)\) that explicitly incorporates label hierarchy information into Transformer\-based text classification\. A pretrained SecureBERT encoder\[[1](https://arxiv.org/html/2607.11994#bib.bib62)\]produces a 768\-dimensional\[CLS\]representation from the input text\. To inject hierarchical knowledge, we introduce a learnable parent\-category embedding that captures coarse\-grained structural relationships among labels\. Both parent categories are projected to a 24\-dimensional dense embedding, trained together with the rest of the network\.

The BERT pooled representation is then fused with the parent embedding, resulting in a unified feature representation\. This represents both semantic content and hierarchical context\. This fused representation is regularized using dropout with a rate of 0\.3 before being passed to a linear classification head that outputs logits over the target classes\.

Algorithm 1Hierarchy\-Aware RoBERTa for Text Classification1:Input tokens

X=\{x1,…,xL\}X=\\\{x\_\{1\},\\dots,x\_\{L\}\\\}, attention mask

MM, parent category IDs

PP, pretrained RoBERTa encoder

ℬ\\mathcal\{B\}, parent embedding matrix

𝐄p\\mathbf\{E\}\_\{p\}
2:Class logits

y^\\hat\{y\}
3:Encode input sequence using RoBERTa:

4:

𝐇←ℬ​\(X,M\)\\mathbf\{H\}\\leftarrow\\mathcal\{B\}\(X,M\)
5:Extract pooled representation:

6:

𝐡c​l​s←𝐇\[CLS\]\\mathbf\{h\}\_\{cls\}\\leftarrow\\mathbf\{H\}\_\{\[\\text\{CLS\}\]\}
7:Obtain parent embedding:

8:

𝐡p←𝐄p​\(P\)\\mathbf\{h\}\_\{p\}\\leftarrow\\mathbf\{E\}\_\{p\}\(P\)
9:Fuse semantic and hierarchical representations:

10:

𝐡f←\[𝐡c​l​s,𝐡p\]\\mathbf\{h\}\_\{f\}\\leftarrow\[\\mathbf\{h\}\_\{cls\},\\mathbf\{h\}\_\{p\}\]
11:Apply dropout:

12:

𝐡~f←Dropout​\(𝐡f\)\\tilde\{\\mathbf\{h\}\}\_\{f\}\\leftarrow\\text\{Dropout\}\(\\mathbf\{h\}\_\{f\}\)
13:Compute class logits:

14:

y^←𝐖​𝐡~f\+𝐛\\hat\{y\}\\leftarrow\\mathbf\{W\}\\tilde\{\\mathbf\{h\}\}\_\{f\}\+\\mathbf\{b\}
15:Optionally, obtain class probabilities and loss \(training\):

16:

𝐩←softmax​\(y^\)\\mathbf\{p\}\\leftarrow\\text\{softmax\}\(\\hat\{y\}\)
17:

ℒ←CrossEntropy​\(𝐩,y\)\\mathcal\{L\}\\leftarrow\\text\{CrossEntropy\}\(\\mathbf\{p\},y\)
18:return

y^\\hat\{y\}

Figure[2](https://arxiv.org/html/2607.11994#S4.F2)illustrates the proposed Hierarchy\-Aware RoBERTa architecture, which combines semantic representations from a pretrained RoBERTa encoder\[[1](https://arxiv.org/html/2607.11994#bib.bib62)\]with learned parent\-label embeddings\. The textual embedding and hierarchical context are fused via concatenation and passed through a dropout\-regularized linear classifier to produce fine\-grained child class predictions\. This design enables hierarchy\-informed classification by jointly leveraging contextual language understanding and structured label information\. To ensure consistency across all models, deep learning models \(CNN, BiGRU, BERT\), and our proposed Hierarchy\-Aware RoBERTa were all trained using the same batch size of 8 and 15 epochs\.

## VExperimental Procedure

### V\-ADataset

The dataset used in this study was sourced from an enhanced version of the MITRE Common Weakness Enumeration \(CWE\) dataset\. The CWE repository is continuously updated and maintained to incorporate newly identified software vulnerabilities and attack patterns\. The dataset111https://cwe\.mitre\.org/data/slices/1000\.htmlcontains944data points with23columns\. For the experiments, we selected three columns \(i\.e\.,CWE\-ID, Name, Weakness Abstraction, Description\)\. TheWeakness Abstractionhas five unique values as presented in Figure[3](https://arxiv.org/html/2607.11994#S5.F3)\. The dataset includes five classes in “W​e​a​k​n​e​s​s​A​b​s​t​r​a​c​t​i​o​nWeaknessAbstraction” column, Base sample of 524, Variant sample of 292, Class sample of 111, Pillar sample of 10, and a Compound sample of 7\.

![Refer to caption](https://arxiv.org/html/2607.11994v1/Figure/dataset.png)Figure 3:Frequency of Weakness Abstraction Labels\.
### V\-BData Processing

In this experiment, all five abstraction classes were retained for classification\. The Name and Description fields were concatenated using a\[SEP\]token after removing noise patterns and trimming whitespace\. Figure[4](https://arxiv.org/html/2607.11994#S5.F4)illustrates a sample of the processed dataset\. Labels fromWeakness Abstractionwere encoded as integers \(0–4\) and tokenized using BertTokenizer\. Data were split 75:25 for training and testing, with all inputs padded or truncated to a maximum sequence length of 128 tokens\.

![Refer to caption](https://arxiv.org/html/2607.11994v1/Figure/process_cwe.png)Figure 4:A sample of CWE dataset after data preprocessing![Refer to caption](https://arxiv.org/html/2607.11994v1/Figure/accuracy.png)Figure 5:Model Accuracy Comparison Across Models

## VIResults

### VI\-AModel Accuracy

Figure[5](https://arxiv.org/html/2607.11994#S5.F5)compares the performance of various machine learning, deep learning, and transformer\-based models in terms of their accuracy as the main metric\. RF achieved a baseline accuracy of 0\.65, improving its performance to 0\.69 with both SMOTE and ADASYN\. SVM performed better than RF with an accuracy of 0\.72, while SMOTE and ADASYN demonstrated the same performance, i\.e\., 0\.71 and 0\.72, respectively\. Such findings suggest that oversampling is weakly helpful with classical models and helps address class imbalance\. Similarly, the vanilla CNN dropped from 0\.71 to 0\.55 with SMOTE and 0\.51 with ADASYN, while BiGRU fell from 0\.70 to 0\.53 and 0\.44, respectively, confirming that oversampling can introduce noise that adversely impacts sequence\-based and convolutional frameworks\. BERT achieved an accuracy of 0\.74, while the proposed Hierarchy\-Aware RoBERTa achieved the highest accuracy of0\.76, demonstrating the advantage of structural priors over synthetic data generation\.

### VI\-BClassification Performance

TABLE I:Per\-Class Performance of Models Across Data Augmentation MethodsTable[I](https://arxiv.org/html/2607.11994#S6.T1)reports the precision, recall, and F1\-score per\-class across all models and augmentation conditions\. The improvement was primarily observed in minority classes\. For example, theClassF1\-score increased from 0\.07 to 0\.42 with SMOTE and 0\.52 with ADASYN for RF\. However,CompoundandPillarremained at 0\.00 F1 across all settings\. SVM showed stronger overall performance with a weighted F1 of 0\.71, maintaining stable scores of 0\.70\-0\.71 under both oversampling methods, indicating robustness on well\-represented classes\.

For deep learning models, CNN achieved a baseline weighted F1 of 0\.68, with strong performance onBase\(F1=0\.79=0\.79\) andVariant\(F1=0\.67=0\.67\)\. However, applying SMOTE and ADASYN reduced the performance to 0\.60 and 0\.56, respectively, withClassF1 dropping to as low as 0\.08 under ADASYN\. BiGRU achieved a baseline weighted F1 of 0\.69, dropping to 0\.58 with SMOTE and 0\.52 with ADASYN, with substantial degradation in minority classes, highlighting the sensitivity of sequence\-based models to oversampling\-induced noise\.

Transformer\-based BERT achieved a strong baseline weighted F1 of 0\.73, driven by high performance onBase\(F1=0\.78=0\.78\) andVariant\(F1=0\.78=0\.78\)\. No augmentation was applied for BERT, as its contextual embeddings were sufficient without synthetic oversampling\. Our Hierarchy\-Aware RoBERTa further improved performance, achieving the highest weighted F1 of0\.76, consistently outperforming all models acrossBase\(F1=0\.81=0\.81\),Class\(F1=0\.60=0\.60\), andVariant\(F1=0\.76=0\.76\)\. In particular,CompoundandPillarremain at 0\.00 F1 in all models, reflecting the persistent challenge of supervised learning under extreme data sparsity\.

## VIIDiscussion

Although data augmentation techniques such as SMOTE and ADASYN are widely used to address class imbalance in classical machine learning, they are not well suited for transformer\-based models\. BERT does not use fixed, low\-dimensional feature vectors, but learns contextualized, high\-dimensional semantic representations directly from raw text, meaning that synthetic interpolation produces samples that lack linguistic coherence and add noise rather than signal\. For CNN and BiGRU models, small perturbations in embedding space correspond to large taxonomic shifts, causing models to overfit to synthetic artifacts and degrading generalization performance on minority and hierarchical classes\.

Furthermore, SMOTE and ADASYN do not conserve the hierarchical relationship that exists within the data\. Classes such asVariantare semantically dependent on their parent classes \(e\.g\.,Base\)\. Synthetic oversampling does not respect such constraints and can produce samples that are inconsistent with their underlying class structure and confuse the model during training\. In the CWE taxonomy, each entry links to aRelated Weaknessvia an explicit parent\-child chain, meaning errors at higher abstraction levels cascade downward to finer\-grained classes, a structural dependency that oversampling methods ignore entirely by treating each sample as an independent entity in a flat feature space\. CWEs are not isolated texts but are organized as a tree structured graph where more abstract weaknesses act as semantic parents to narrower ones, and interpolating between embeddings from different parent nodes produces representations of non\-existent or indistinct weaknesses, giving statistically balanced but structurally invalid embeddings\.

Our Hierarchy\-Aware RoBERTa model explicitly incorporates structural knowledge by introducing a learnable parent embedding\. In the CWE taxonomy, weakness abstractions are strictly hierarchical\. For instance, a"Variant"class is semantically and structurally dependent on its corresponding"Base"parent\. To model this dependency, we formulate the classification task not as an independent predictionP​\(Yc​h​i​l​d\|Xt​e​x​t\)P\(Y\_\{child\}\|X\_\{text\}\), but as a conditional probability problemP​\(Yc​h​i​l​d\|Xt​e​x​t,Yp​a​r​e​n​t\)P\(Y\_\{child\}\|X\_\{text\},Y\_\{parent\}\)\. We utilize"Parent ID"as a structural prior to guide the model to learn that if the parent is aBasetype, the child is likely aVariant\. We define two input vectors: Text Vector \(Vt​e​x​tV\_\{text\}\): The output from BERT \(768 dimensions\)\. Parent Vector \(Vg​r​a​p​hV\_\{graph\}\): A learnable embedding for the parent ID \(24 dimensions\)\. The model fuses them via concatenation:Vf​i​n​a​l=Concat​\(Vt​e​x​t,Vg​r​a​p​h\)V\_\{final\}=\\text\{Concat\}\(V\_\{text\},V\_\{graph\}\),Vf​i​n​a​l∈ℝ768\+24V\_\{final\}\\in\\mathbb\{R\}^\{768\+24\}\. The classifier then predicts based on this combined knowledge:yp​r​e​d=Softmax​\(W⋅Vf​i​n​a​l\+b\)y\_\{pred\}=\\text\{Softmax\}\(W\\cdot V\_\{final\}\+b\)\.

### VII\-ALimitations of the Hierarchy\-Aware RoBERTa Model

Despite its improved performance, the proposed Hierarchy\-Aware RoBERTa model has several limitations\. First, the architecture assumes parent\-level metadata availability during inference, though this is consistent with coarse\-to\-fine vulnerability pipelines where high\-level categories are readily identifiable via static analysis tools \(SAST\) or manual triage\. Second, the hierarchy is treated as fixed, and the model may not generalize to new or rearranged nodes without retraining, as the CWE taxonomy is subject to continual updates\. Third, only direct parent information is captured, disregarding multi\-hop ancestors, sibling relationships, and tree depth, which may limit discrimination among deeply nested classes such asCompoundandPillar\. When training data is virtually absent, hierarchy awareness alone is insufficient, suggesting that weak supervision or external knowledge may be necessary\. Finally, the learnable parent embeddings add modest parameter overhead that could hinder scaling to large taxonomies, and the approach has been evaluated only within the CWE domain, leaving generalizability to other hierarchical tasks empirically unvalidated\.

## VIIIConclusion

This study examined the effectiveness of oversampling techniques and hierarchy\-aware modeling to address class imbalance in CWE vulnerability classification\. Through extensive experimentation, we demonstrated that commonly used oversampling methods, such as SMOTE and ADASYN, provide limited benefits for machine learning models \(i\.e\., RF and SVM\) and consistently degrade performance for deep learning models \(i\.e\., CNN and BiGRU\)\. These methods generate synthetic samples through linear interpolation in embedding space, an assumption that does not hold for contextualized text representations and often disrupts the semantic and hierarchical integrity of CWE labels\.

To overcome these limitations, we proposed a Hierarchy\-Aware RoBERTa model that explicitly integrates parent–child relationships from the CWE taxonomy into the classification process\. By incorporating learnable parent embeddings alongside contextual language representations, the model enforces structural consistency and improves discrimination among minority and semantically dependent classes\. The proposed model achieves a weighted F1\-score of0\.760\.76without data augmentation, outperforming all baselines, including BERT, which achieved0\.740\.74, demonstrating that structural priors are more effective than synthetic data generation in hierarchical imbalanced cybersecurity datasets\.

Despite these gains, extremely sparse categories such asCompoundandPillarremain difficult to predict, highlighting the inherent limitations of supervised learning when training data is severely limited\. Future research may extend the hierarchy\-aware framework to incorporate multi\-hop ancestry and graph neural networks over the full CWE hierarchy, explore weak supervision and label propagation for extremely sparse classes, and evaluate the proposed approach across other hierarchical cybersecurity datasets to establish its generalizability beyond the CWE domain\.

## Acknowledgment

This research is partially supported by the U\.S\. National Science Foundation \(Award \#: 2319802\)\.

## References

- \[1\]E\. Aghaei, X\. Niu, W\. Shadid, and E\. Al\-Shaer\(2022\)Securebert: a domain\-specific language model for cybersecurity\.InInternational Conference on Security and Privacy in Communication Systems,pp\. 39–56\.Cited by:[§III\-C1](https://arxiv.org/html/2607.11994#S3.SS3.SSS1.p1.1),[§IV\-C](https://arxiv.org/html/2607.11994#S4.SS3.p1.1),[§IV\-C](https://arxiv.org/html/2607.11994#S4.SS3.p3.1)\.
- \[2\]L\. Breiman\(2001\)Random forests\.Machine learning45\(1\),pp\. 5–32\.Cited by:[§II\-B](https://arxiv.org/html/2607.11994#S2.SS2.p1.1),[§III\-A1](https://arxiv.org/html/2607.11994#S3.SS1.SSS1.p1.1)\.
- \[3\]N\. V\. Chawla, K\. W\. Bowyer, L\. O\. Hall, and W\. P\. Kegelmeyer\(2002\)SMOTE: synthetic minority over\-sampling technique\.Journal of Artificial Intelligence Research16,pp\. 321–357\.External Links:[Link](https://scholar.google.com/scholar?q=SMOTE+Synthetic+Minority+Oversampling+Technique+Chawla)Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.11994#S2.SS3.p1.1)\.
- \[4\]B\. Chhetri and A\. S\. Namin\(2025\)The application of transformer\-based models for predicting consequences of cyber attacks\.In2025 IEEE 49th Annual Computers, Software, and Applications Conference \(COMPSAC\),pp\. 523–532\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.11994#S2.SS1.p1.1)\.
- \[5\]J\. Chung, C\. Gulcehre, K\. Cho, and Y\. Bengio\(2014\)Empirical evaluation of gated recurrent neural networks on sequence modeling\.arXiv preprint arXiv:1412\.3555\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p2.1),[§III\-B2](https://arxiv.org/html/2607.11994#S3.SS2.SSS2.p1.1)\.
- \[6\]C\. Contreras, H\. Dokic, Z\. Huang, D\. Stan Raicu, J\. Furst, and R\. Tchoua\(2023\)Multiclass classification of software vulnerabilities with deep learning\.InProceedings of the 2023 15th International Conference on Machine Learning and Computing,pp\. 134–140\.Cited by:[§II\-A](https://arxiv.org/html/2607.11994#S2.SS1.p1.1)\.
- \[7\]S\. S\. Das, E\. Serra, M\. Halappanavar, A\. Pothen, and E\. Al\-Shaer\(2021\)V2w\-bert: a framework for effective hierarchical multiclass classification of software vulnerabilities\.In2021 IEEE 8th International Conference on Data Science and Advanced Analytics \(DSAA\),pp\. 1–12\.Cited by:[§II\-B](https://arxiv.org/html/2607.11994#S2.SS2.p1.1)\.
- \[8\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p2.1),[§III\-C1](https://arxiv.org/html/2607.11994#S3.SS3.SSS1.p1.1)\.
- \[9\]H\. He, Y\. Bai, E\. A\. Garcia, and S\. Li\(2008\)ADASYN: adaptive synthetic sampling approach for imbalanced learning\.InIEEE International Joint Conference on Neural Networks \(IJCNN\),pp\. 1322–1328\.External Links:[Link](https://scholar.google.com/scholar?q=ADASYN+Adaptive+Synthetic+Sampling+He)Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.11994#S2.SS3.p1.1)\.
- \[10\]M\. A\. Hearst, S\. T\. Dumais, E\. Osuna, J\. Platt, and B\. Scholkopf\(1998\)Support vector machines\.IEEE Intelligent Systems and their applications13\(4\),pp\. 18–28\.Cited by:[§II\-B](https://arxiv.org/html/2607.11994#S2.SS2.p1.1),[§III\-A2](https://arxiv.org/html/2607.11994#S3.SS1.SSS2.p1.1)\.
- \[11\]R\. Jiang, M\. Liu, H\. Liu, and Y\. Yu\(2025\)Hierarchy\-consistent learning and adaptive loss balancing for hierarchical multi\-label classification\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 1190–1199\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p2.1),[§I](https://arxiv.org/html/2607.11994#S1.p3.1)\.
- \[12\]S\. Kishanthan and A\. Hevapathige\(2025\)Deep learning meets oversampling: a learning framework to handle imbalanced classification\.International Journal of Information Technology,pp\. 1–13\.Cited by:[§II\-C](https://arxiv.org/html/2607.11994#S2.SS3.p1.1)\.
- \[13\]K\. Kota, A\. Manjunatha,et al\.\(2024\)CWE prediction using cve description\-the semantic similarity approach\.Procedia Computer Science235,pp\. 1167–1178\.External Links:[Document](https://dx.doi.org/10.1016/j.procs.2024.09.234)Cited by:[§II\-B](https://arxiv.org/html/2607.11994#S2.SS2.p1.1)\.
- \[14\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)Roberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p2.1),[§III\-C1](https://arxiv.org/html/2607.11994#S3.SS3.SSS1.p1.1)\.
- \[15\]MITRE\(2025\)CWE\- home\.Note:[https://cwe\.mitre\.org/about/index](https://cwe.mitre.org/about/index)Accessed: Dec\. 14, 2025Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p1.1)\.
- \[16\]H\. Turtiainen and A\. Costin\(2024\)VulnBERTa: on automating CWE weakness assignment and improving the quality of cybersecurity CVE vulnerabilities through ml/nlp\.In2024 IEEE European Symposium on Security and Privacy Workshops \(EuroS&PW\),pp\. 618–625\.External Links:[Document](https://dx.doi.org/10.1109/EuroSPW62116.2024.00082)Cited by:[§II\-A](https://arxiv.org/html/2607.11994#S2.SS1.p1.1)\.
- \[17\]R\. Yamashita, M\. Nishio, R\. K\. G\. Do, and K\. Togashi\(2018\)Convolutional neural networks: an overview and application in radiology\.Insights into imaging9\(4\),pp\. 611–629\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p2.1),[§III\-B1](https://arxiv.org/html/2607.11994#S3.SS2.SSS1.p1.1)\.
- \[18\]C\. Yang, E\. A\. Fridgeirsson, J\. A\. Kors, J\. M\. Reps, and P\. R\. Rijnbeek\(2024\)Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data\.Journal of Big Data11\(1\),pp\. 7\.Cited by:[§I](https://arxiv.org/html/2607.11994#S1.p3.1)\.