LLM-Based Weak Annotation for Cybersecurity Tweet Classification: A Teacher-Student Framework

# LLM-Based Weak Annotation for Cybersecurity Tweet Classification: A Teacher–Student Framework
Source: [https://arxiv.org/html/2610.21254](https://arxiv.org/html/2610.21254)
Ferry Afit Kurniawan, Kazushi Okamoto, Koki Karube, Kei Harada, Atsushi ShibataAffiliation:Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro\-CommunicationsAffiliation:1\-5\-1 Chofugaoka, Chofu, Tokyo 182\-8585, JapanAffiliation:\{ferry\.afit, kazushi, karubekoki, harada, shibata\-atsushi\}@uec\.ac\.jp

###### Abstract

Cyber threat intelligence \(CTI\) relies on the timely identification of emerging threats, enabling organizations and security analysts to support situational awareness and decision\-making\. Social media platforms such as X \(formerly Twitter\) have become valuable CTI sources because vulnerabilities and attack activities are often discussed there before appearing in formal reports\. However, the rapid growth of cybersecurity discussions on social media has created more content than domain experts can realistically review and annotate, making labeled data a major bottleneck for developing supervised learning approaches\. While large language models \(LLMs\) have shown promise as weak annotators in other domains, their effectiveness for cybersecurity tweet classification and downstream training of domain\-specific pre\-trained language models \(PLMs\) remains underexplored\. To address this challenge, we propose a Teacher–Student weak supervision framework in which a selected LLM serves as a Teacher to generate weak labels for unlabeled cybersecurity tweets, enabling the training of cybersecurity\-specific PLMs without large\-scale manual annotation\. Experimental results show that PLMs trained solely on LLM\-generated labels consistently achieve F1 scores above 0\.92, remaining within 3 percentage points of fully human\-supervised models across all evaluated architectures\. These findings suggest that LLM\-based weak annotation offers a practical alternative for constructing cybersecurity\-specific training data when expert annotation resources are limited\.

###### Index Terms:

Cyber threat intelligence, weak annotation, large language models, teacher\-student framework, tweet classification, pre\-trained language models, weak supervision\.

††aftertitle:© 2026 IEEE\. Personal use of this material is permitted\. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works\.## IIntroduction

Cyber threat intelligence \(CTI\) supports cybersecurity decision\-making by providing actionable information about emerging threats, vulnerabilities, and adversarial activity at strategic, operational, and tactical levels\[[1](https://arxiv.org/html/2610.21254#bib.bib1),[2](https://arxiv.org/html/2610.21254#bib.bib2)\]\. Modern CTI systems combine heterogeneous information sources, knowledge bases, and analytical tools to transform raw threat data into insights that support security analysts and organizational decisions\[[1](https://arxiv.org/html/2610.21254#bib.bib1)\]\. Among these sources, social media platforms such as X \(formerly Twitter\) have become valuable channels for collecting timely CTI\. Cybersecurity researchers, vendors, and practitioners often share newly discovered vulnerabilities, malware campaigns, exploit activities, and indicators of compromise \(IOCs\) on these platforms before the same information appears in formal advisories or public databases\[[3](https://arxiv.org/html/2610.21254#bib.bib3),[4](https://arxiv.org/html/2610.21254#bib.bib4),[5](https://arxiv.org/html/2610.21254#bib.bib5),[6](https://arxiv.org/html/2610.21254#bib.bib6),[7](https://arxiv.org/html/2610.21254#bib.bib7)\]\. Automated analysis of social media streams can therefore complement traditional threat intelligence pipelines and improve cyber situational awareness\.

Despite this potential, cybersecurity\-related tweets are noisy, brief, and produced at a volume that far exceeds what experts can realistically review and annotate manually\[[5](https://arxiv.org/html/2610.21254#bib.bib5),[7](https://arxiv.org/html/2610.21254#bib.bib7)\]\. This creates a bottleneck for supervised learning approaches, which depend on high\-quality labeled data for training\. Although numerous models have been proposed for cybersecurity tweet classification and related CTI tasks, their effectiveness depends on carefully curated annotated datasets\[[3](https://arxiv.org/html/2610.21254#bib.bib3),[4](https://arxiv.org/html/2610.21254#bib.bib4),[8](https://arxiv.org/html/2610.21254#bib.bib8),[9](https://arxiv.org/html/2610.21254#bib.bib9),[10](https://arxiv.org/html/2610.21254#bib.bib10),[13](https://arxiv.org/html/2610.21254#bib.bib13)\]\.

Weak supervision offers a promising way to reduce the reliance on exhaustive manual annotation\. Additionally, recent advances in natural language processing \(NLP\) have demonstrated that large language models \(LLMs\) can generate weak annotations for training downstream classifiers in a variety of text classification tasks\[[12](https://arxiv.org/html/2610.21254#bib.bib12),[17](https://arxiv.org/html/2610.21254#bib.bib17),[18](https://arxiv.org/html/2610.21254#bib.bib18),[19](https://arxiv.org/html/2610.21254#bib.bib19),[20](https://arxiv.org/html/2610.21254#bib.bib20)\]\. However, the reliability of LLM\-generated weak labels for cybersecurity tweet classification, and their usefulness for training cybersecurity\-specific pre\-trained language models \(PLMs\), remain insufficiently explored\.

To address this gap, we propose aTeacher\-Student weak supervision frameworkfor cybersecurity tweet classification\. First, multiple candidate LLMs are evaluated as Teacher models\. Next, the most reliable model is selected to annotate a large corpus of unlabeled cybersecurity tweets\. Finally, these weakly labeled data are used to fine\-tune cybersecurity\-specific PLMs, and their performance is compared under human\-only, LLM\-only, and hybrid supervision settings\. This design enables us to examine both the annotation quality of LLM Teachers and the downstream value of their weak labels for domain\-specific Student models\. Accordingly, this study is guided by three research questions:

RQ1Which candidate LLM provides the most reliable weak annotations for cybersecurity tweets?

RQ2Can LLM\-generated weak labels effectively supervise the training of cybersecurity\-specific PLMs?

RQ3How do LLM\-only, human\-only, and hybrid supervision compare in downstream classification performance?

The main contributions of this work are summarized as follows:

- •We systematically evaluate three LLMs as weak annotators for cybersecurity tweet classification across public benchmark datasets and a newly collected gold\-standard dataset\.
- •We introduce a scalable Teacher\-Student pipeline that uses LLM\-generated weak labels to construct a training set for cybersecurity\-specific PLMs, reducing the need for extensive manual annotation\.
- •Across four cybersecurity\-specific PLMs, we show that LLM\-only training recovers approximately 98% of human\-only F1 on average, indicating that weak annotation can provide competitive downstream performance under limited annotation resources\.

## IIRelated Work

### II\-ACybersecurity Tweet Classification

Initial work on cybersecurity tweet classification centered on automatically identifying cybersecurity\-related content in social media streams\. Dionísio et al\.\[[3](https://arxiv.org/html/2610.21254#bib.bib3)\]applied deep neural networks to cyber threat detection on Twitter, while Alves et al\.\[[4](https://arxiv.org/html/2610.21254#bib.bib4)\]investigated large\-scale tweet processing for cybersecurity threat awareness\. Subsequent studies explored novelty detection\[[5](https://arxiv.org/html/2610.21254#bib.bib5)\], contrastive word embeddings\[[8](https://arxiv.org/html/2610.21254#bib.bib8)\], and hybrid IDCNN–BiLSTM architectures\[[9](https://arxiv.org/html/2610.21254#bib.bib9)\]to improve classification performance\. More recently, transformer\-based approaches have become increasingly prominent\. Marinho and Holanda\[[13](https://arxiv.org/html/2610.21254#bib.bib13)\]applied NLP techniques to automated CTI, whereas Bayer et al\.\[[14](https://arxiv.org/html/2610.21254#bib.bib14)\], Kebeli and Kwon\[[15](https://arxiv.org/html/2610.21254#bib.bib15)\], and You et al\.\[[16](https://arxiv.org/html/2610.21254#bib.bib16)\]explored fine\-tuning, self\-training, and prompt\-based learning to reduce dependence on large labeled datasets while maintaining competitive performance\. These studies demonstrate the effectiveness of modern NLP methods for CTI\-oriented tweet classification, but they still primarily focus on model design or task\-specific supervision rather than LLM\-generated weak labels for cybersecurity\-specific PLMs\.

### II\-BLLM\-Based Weak Supervision

Recent advances in LLMs have enabled their use as automatic annotators for generating weak labels in NLP tasks\. Gilardi et al\.\[[17](https://arxiv.org/html/2610.21254#bib.bib17)\]reported that ChatGPT can achieve annotation quality comparable to or better than that of human crowd workers across several classification tasks\. He et al\. proposed AnnoLLM\[[18](https://arxiv.org/html/2610.21254#bib.bib18)\], which improves annotation quality through explanation\-guided prompting, while Pangakis and Wolken\[[12](https://arxiv.org/html/2610.21254#bib.bib12)\]demonstrated that LLM\-generated labels can effectively supervise downstream classifiers through knowledge distillation\. Similarly, Bansal and Sharma\[[19](https://arxiv.org/html/2610.21254#bib.bib19)\]showed that LLM\-based annotation can improve model generalization at relatively low labeling cost\. Beyond fully automated annotation, hybrid annotation methods have also attracted increasing attention\. MEGAnno\+\[[20](https://arxiv.org/html/2610.21254#bib.bib20)\]provides an interactive human–LLM collaborative annotation environment, and Wang et al\.\[[21](https://arxiv.org/html/2610.21254#bib.bib21)\]proposed verification mechanisms that combine LLM\-generated labels with selective human review to improve reliability and efficiency\.

### II\-CResearch Gap and Positioning

Although prior studies have advanced cybersecurity tweet classification and LLM\-based weak annotation, the intersection of these two research directions remains largely unexplored\. Existing cybersecurity studies mainly focus on developing classification architectures or reducing supervision through self\-training and prompt\-based learning, whereas LLM\-based annotation has been evaluated predominantly on general\-domain NLP tasks\. Consequently, it remains unclear whether LLM\-generated weak labels are reliable enough to train cybersecurity\-specific PLMs for CTI\-oriented tweet classification\. Recent Teacher–Student frameworks have demonstrated the feasibility of training classifiers from LLM\-generated labels without manual annotation\[[22](https://arxiv.org/html/2610.21254#bib.bib22)\]; however, their applicability to cybersecurity\-specific PLMs and to supervision regimes that combine human and LLM labels has not been systematically investigated\. This work addresses this gap by evaluating multiple LLMs as Teacher models and assessing whether their weak labels can effectively train cybersecurity\-specific PLMs under human\-only, LLM\-only, and hybrid supervision settings\.

Table[I](https://arxiv.org/html/2610.21254#S2.T1)summarizes representative prior studies and positions the proposed framework relative to existing research\.

TABLE I:Comparison of representative studies and the proposed work\.

## IIIMethodology

### III\-AOverall Framework

The framework consists of five sequential stages, as illustrated in Fig\.[1](https://arxiv.org/html/2610.21254#S3.F1)\. First, data sources are prepared by combining public benchmark datasets with a newly collected cybersecurity tweet corpus, including a manually annotated subset for evaluation and supervised baselines\. Second, multiple candidate LLMs are evaluated as potential Teacher models, and the most reliable model is selected based on annotation performance\. Third, the selected Teacher assigns weak labels to previously unlabeled cybersecurity tweets, producing a large weakly labeled corpus\. Fourth, this corpus is used to fine\-tune multiple cybersecurity\-specific PLMs, which serve as Student models\. Finally, the resulting classifiers are evaluated and compared across human\-only, LLM\-only, and hybrid supervision settings\.

![Refer to caption](https://arxiv.org/html/2610.21254v1/figure/pipeline.jpeg)Fig\. 1:Teacher–Student framework
### III\-BDatasets

#### III\-B1Data Collection

To construct a new cybersecurity tweet corpus, we collected tweets from 112 cybersecurity\-related X accounts between November 2024 and January 2025\. The selected accounts represent diverse and reputable sources of CTI, including 49 cybersecurity researchers, 39 security vendors, 7 cybersecurity organizations and agencies, and 17 cybersecurity news outlets\. A total of 17,581 tweets were collected\. After removing non\-English tweets and duplicates and normalizing the text, 13,451 unique English\-language tweets remained\. To build a gold\-standard benchmark, we selected a stratified sample of 2,000 tweets for manual annotation\. Each tweet was independently labeled by two annotators with expertise in cybersecurity, following predefined annotation guidelines\. Inter\-annotator agreement was measured using Cohen’s Kappa \(κ=0\.89\\kappa=0\.89\), which indicates high agreement\. To ensure high label quality, only tweets for which both annotators assigned the same label were retained\. As a result, 103 tweets with annotation disagreements were excluded, yielding a final gold\-standard dataset of 1,897 tweets\.

#### III\-B2Data Splitting

Among the annotated tweets, 400 were reserved as an independent held\-out test set and were not used for Teacher selection, weak annotation, or Student model training\. The remaining 1,497 manually labeled tweets were used in the human\-only and hybrid supervision settings\. For Teacher selection, 400 tweets were sampled from this 1,497\-tweet pool\. The 11,451 tweets outside the annotated subset remained unlabeled and were annotated automatically by the selected Teacher model\. These weakly labeled tweets were subsequently used to train the Student models\.

#### III\-B3Public Benchmark Datasets

In addition to the newly collected dataset, we used the publicly available Behzadan\[[23](https://arxiv.org/html/2610.21254#bib.bib23)\]and CybAttT\[[24](https://arxiv.org/html/2610.21254#bib.bib24)\]datasets during Teacher model selection\. Evaluating the candidate Teachers on multiple datasets helps assess performance across different data distributions and reduces the risk of selecting a model that performs well on only a single benchmark\.

#### III\-B4Label Harmonization

The three datasets used in this study were originally annotated using different label schemas\. To enable a consistent evaluation of Teacher models and downstream Student training, all datasets were converted into a unified binary classification task consisting ofRelevantandIrrelevantclasses\. Tweets were annotated asRelevantif they contained concrete cyber threat information, such as CVEs and other vulnerabilities, exploits, malware or ransomware, active attacks, security incidents, IOCs, or attributed threat actors and campaigns\. Tweets that did not contain concrete threat information were labeled asIrrelevant\. The newly collected dataset was already annotated using the binary labelsRelevantandIrrelevantand therefore required no label harmonization\. For the CybAttT dataset, bothNormal NewsandHigh Risk Newswere mapped to theRelevantclass, whereasNot Newswas mapped to theIrrelevantclass\. For the Behzadan dataset, theThreatcategory was mapped to theRelevantclass, while theBusinessandIrrelevantcategories were merged into theIrrelevantclass\. Table[II](https://arxiv.org/html/2610.21254#S3.T2)summarizes the label harmonization process across the three datasets\.

TABLE II:Label harmonization across datasets\.

### III\-CTeacher Model Selection

We evaluated three candidate LLMs: GPT\-4\.1\-nano, GPT\-4o\-mini, and Llama\-3\.3\-70B Instruct\. The evaluation was conducted using manually labeled samples drawn from the three benchmark datasets described previously\. Specifically, 150 Relevant and 150 Irrelevant tweets were sampled from each of Behzadan and CybAttT, and 400 tweets were sampled from the gold\-data pool described above, resulting in 1,000 evaluation samples\. All candidate models were evaluated under the same zero\-shot prompting setting\. For each tweet, the model was asked to determine whether the content wasRelevantorIrrelevantto cybersecurity\. The predicted labels were then compared with the human\-annotated ground truth\. The complete prompt template is shown in Fig\.[2](https://arxiv.org/html/2610.21254#S3.F2)\.

Zero\-shot prompt \(universal across datasets,T=0T=0\): Role:Senior Cyber Threat Intelligence Analyst\. Task:Classify the tweet as Relevant \(1\) or Irrelevant \(0\)\.A tweet is Relevant \(1\) if it contains CONCRETE information about a cyber threat – a specific CVE or vulnerability, a named exploit, a named malware family, an active attack or security incident with a named victim or target, an indicator of compromise \(IP, hash, domain, file path\), or an attributed threat actor or campaign\.A tweet is Irrelevant \(0\) if it contains NO concrete threat instance – including general cybersecurity discussion, security news commentary, marketing, job postings, opinions, educational content, policy discussions, software releases, or statistical trends about cybersecurity without naming a specific incident\.Operative test: does this tweet reference a SPECIFIC, NAMED threat instance that a CTI analyst could act on? If not, label 0\.When genuinely uncertain, default to 0\.

Constraint:Output ONLY the digit 0 or 1\.

Fig\. 2:Universal zero\-shot annotation prompt, used identically across all Teacher LLM candidates and all three datasets\.
### III\-DWeak Annotation

The selected Teacher model was applied to 11,451 unlabeled cybersecurity tweets to generate weak labels for downstream model training\. The Teacher produced 7,275Relevant\(63\.5%\) and 4,176Irrelevantlabels \(36\.5%\)\. The resulting weakly labeled corpus served as the training data for the LLM\-only setting and as an additional supervision source in the hybrid setting\. Thus, the downstream experiments evaluate the utility of the raw Teacher\-generated labels without manual correction\.

### III\-EStudent PLM Training

Four cybersecurity\-specific PLMs were selected as Student models: CTI\-BERT\[[11](https://arxiv.org/html/2610.21254#bib.bib11)\], CySecBERT\[[25](https://arxiv.org/html/2610.21254#bib.bib25)\], SecureBERT\-2\.0\[[26](https://arxiv.org/html/2610.21254#bib.bib26)\], and SecRoBERTa\[[27](https://arxiv.org/html/2610.21254#bib.bib27)\]\. These models were chosen because they were pre\-trained on cybersecurity\-related corpora and represent diverse architectures and training strategies within the cybersecurity NLP domain\. Using multiple Student models reduces the likelihood that the findings are tied to a particular architecture and provides a broader assessment of the proposed framework\.

- •Human\-only: trained using the 1,497 manually annotated tweets\.
- •LLM\-only: trained using the 11,451 weak labels generated by the selected Teacher model\.
- •Hybrid: trained using the combination of 1,497 manually annotated tweets and 11,451 LLM\-generated labels\.

### III\-FEvaluation Procedure

Teacher model performance was measured on the 1,000\-tweet evaluation set described in Sec\. III\-C using accuracy, precision, recall, and F1\-score\. The model achieving the strongest overall F1 across the three datasets is selected as the Teacher and used for all subsequent weak annotation\. Student PLM performance was measured on the 400\-tweet held\-out test set, which was set aside before any training and never used for model selection\. For comparisons among supervision regimes, we applied an exact permutation Friedman test to the five paired seed\-level F1\-scores for each PLM\. Pairwise Wilcoxon signed\-rank comparisons with Holm correction were reported descriptively because five paired seeds limit the attainable exactpp\-values\.

## IVExperiment and Results

### IV\-ATeacher Model Selection

Table[III](https://arxiv.org/html/2610.21254#S4.T3)reports per\-dataset performance\. On CybAttT, all three models achieved F1 scores above 0\.82, while F1 scores decreased for all three candidates on Behzadan\. This decrease was mainly associated with lower recall, suggesting that some Behzadan tweets mapped as Relevant after label harmonization may have fallen outside the concrete cyber threat information criterion used in our prompt\. On the gold\-standard evaluation sample, Llama\-3\.3\-70B achieved the highest F1 score \(0\.9225\), closely followed by GPT\-4\.1\-nano \(0\.9137\)\.

TABLE III:Zero\-Shot LLM Annotation Quality Per DatasetTABLE IV:PLM Classification PerformanceTABLE V:Size\-Matched LLM\-only AblationAcross the full 1,000\-tweet evaluation set, Llama\-3\.3\-70B achieved the highest F1 score \(0\.8436\), followed by GPT\-4\.1\-nano \(0\.8312\) and GPT\-4o\-mini \(0\.7758\)\. McNemar’s test confirmed that both top\-performing models significantly outperformed GPT\-4o\-mini \(χ2≥9\.51\\chi^\{2\}\\geq 9\.51,p<0\.01p<0\.01for both comparisons\)\. Although the difference between Llama\-3\.3\-70B and GPT\-4\.1\-nano was not significant \(χ2=0\.01\\chi^\{2\}=0\.01,p=0\.92p=0\.92\), Llama\-3\.3\-70B was selected as the Teacher based on its highest F1 score\. It also achieved higher recall on Our Data \(0\.9393 vs\. 0\.8785\), which is desirable for CTI pre\-filtering\. However, GPT\-4\.1\-nano incurred a lower annotation cost than Llama\-3\.3\-70B \($0\.0100 versus $0\.0158\) for 400 tweets, indicating a cost–performance trade\-off\.

### IV\-BStudent PLM Classification

Table[IV](https://arxiv.org/html/2610.21254#S4.T4)reports PLM performance on the 400\-tweet held\-out test set\. Each training configuration was run five times with different random seeds; results are reported as mean±\\pmstandard deviation across the five runs\. Across all four PLMs, LLM\-only training recovered approximately 98% of the human\-only F1 score on average \(range 97\.6%–98\.8%\)\. The largest mean F1 gap was observed for CTI\-BERT \(0\.9543 versus 0\.9312\)\. Hybrid training achieved a higher mean F1 than LLM\-only training for all PLMs and achieved the highest mean F1 for SecRoBERTa \(0\.9392\)\.

Exact permutation Friedman tests indicated significant overall differences among the supervision regimes for all PLMs \(Table IV:Q=6\.4Q=6\.4–8\.48\.4,p=0\.008p=0\.008–0\.0390\.039; Table V:Q=8\.4Q=8\.4–10\.010\.0,p<0\.009p<0\.009\)\. With five paired seeds, the smallest attainable two\-sided exact Wilcoxonpp\-value is 0\.0625\. No Holm\-adjusted pairwise comparison can therefore reach the 0\.05 threshold\. Pairwise differences are consequently interpreted descriptively based on mean F1\.

### IV\-CSize\-matched LLM\-only Ablation

To disentangle the contribution of labelquantityfrom labelquality, each PLM was also trained on a size\-matched LLM\-only subset ofn=1,497n=1\{,\}497tweets\. This subset matched the size of the human\-only pool and was drawn from the full LLM\-only corpus using stratified sampling\. The subset was drawn once and held fixed across seeds\. Thus, the reported variability does not reflect variation in subset selection\. Table[V](https://arxiv.org/html/2610.21254#S4.T5)compares these results with those obtained from full LLM\-only and human\-only training\. When restricted to the same number of labels \(n=1,497n=1\{,\}497\), human\-only training yielded 2\.9–3\.3 percentage points higher mean F1 than size\-matched LLM\-only training across all four models, suggesting that manual labels provided a stronger supervision signal at equal scale\. The full LLM\-only corpus \(11,451 tweets; 7\.6×\\timesthe human\-only pool\) reduced this mean F1 gap to 1\.1–2\.3 percentage points\. The complete workflow, comprising evaluation of three candidate Teachers on the 1,000\-tweet evaluation set and weak annotation of the 11,451 unlabeled tweets, incurred total API charges of less than US$0\.70\.

### IV\-DDiscussion

#### IV\-D1RQ1: Which candidate LLM provides the most reliable weak annotations for cybersecurity tweets?

Llama\-3\.3\-70B was selected as the Teacher model because it achieved the highest overall F1\-score and the highest recall on the gold\-standard dataset\. Although its F1\-score was not significantly different from that of GPT\-4\.1\-nano \(p=0\.92p=0\.92\), its higher recall is particularly desirable for CTI pipelines, where a relevance classifier serves as the initial filtering stage and missing a genuine threat is generally more costly than flagging additional false positives for review\.

#### IV\-D2RQ2: Can LLM\-generated weak labels effectively supervise the training of cybersecurity\-specific PLMs?

At matched training\-set size, human\-only training yielded higher mean F1 across all PLMs\. The gap narrowed when the full weakly labeled corpus was used, suggesting that the larger training set contributed to the recovery\. LLM\-generated weak labels therefore provide an effective supervision signal, with scale rather than per\-label accuracy driving the improvement\.

#### IV\-D3RQ3: How do LLM\-only, human\-only, and hybrid supervision compare in downstream classification performance?

The three supervision settings revealed a trade\-off between annotation quality and training data quantity\. Human\-only supervision yielded the highest mean F1 for most PLMs\. Hybrid supervision showed a higher mean F1 than LLM\-only training for all PLMs but rarely surpassed human\-only training, suggesting that additional weak labels may also introduce label noise\. Because the Teacher assigns class labels rather than generating text, this noise takes the form of unsupported label assignments rather than fabricated content\.

### IV\-ELimitations

This study has several limitations that provide opportunities for future work\. First, the proposed framework focuses on binary classification, distinguishingRelevantfromIrrelevant\. This design was chosen because relevance filtering is typically the first stage of a CTI pipeline before more specific analyses, such as topic modeling or event detection\. Second, only English\-language tweets were included in this study\. Evaluating the framework on multilingual datasets would provide broader coverage, since threat intelligence is also shared in many other languages\. Finally, the newly collected dataset was constructed from 112 curated cybersecurity\-related X accounts to reduce noise and improve the efficiency of dataset construction\. However, it may not fully capture cybersecurity discussions from newly emerging sources\. Future research should explore broader collection strategies while maintaining data quality\.

## VConclusion

In this study, we proposed and evaluated a Teacher\-Student weak supervision framework for cybersecurity tweet classification, in which an LLM serves as the Teacher to generate weak labels for training domain\-specific PLMs\. The experimental results showed that Llama\-3\.3\-70B was the most reliable Teacher model, while Student models trained solely on LLM\-generated labels achieved performance close to that of fully supervised models\. Additional experiments suggested that scaling training data can partially offset lower label quality\. These findings demonstrate that LLM\-based weak annotation offers a practical approach to reducing manual annotation effort while maintaining competitive downstream performance\. For cybersecurity applications, where expert annotation is both expensive and time\-consuming, the proposed framework provides an effective alternative for constructing large\-scale training datasets with limited human involvement\. Future work should extend this framework by integrating topic modeling and event detection to construct structured CTI events from weakly labeled social media streams\.

## References

- \[1\]N\. Sun, M\. Ding, J\. Jiang, W\. Xu, X\. Mo, Y\. Tai, and J\. Zhang, “Cyber Threat Intelligence Mining for Proactive Cybersecurity Defense: A Survey and New Perspectives,”*IEEE Commun\. Surv\. Tutor\.*, vol\. 25, no\. 3, pp\. 1748–1774, 2023, doi: 10\.1109/COMST\.2023\.3273282\.
- \[2\]M\. Kriesten, M\. Thinyane, and D\. Ormrod, “Measures, Metrics, and a Scale for Appraisal of Cyber Threat Intelligence\-Informed Decision\-Making,” in*Proc\. 24th Eur\. Conf\. Cyber Warfare and Security \(ECCWS\)*, pp\.284–292, 2025, doi: 10\.34190/eccws\.24\.1\.3421\.
- \[3\]N\. Dionísio, F\. Alves, P\. M\. Ferreira, and A\. Bessani, “Cyberthreat Detection from Twitter using Deep Neural Networks,” in*Proc\. 2019 Int\. Joint Conf\. Neural Netw\. \(IJCNN\)*, pp\. 1–8, 2019, doi: 10\.1109/ijcnn\.2019\.8852475\.
- \[4\]F\. Alves, A\. Bettini, P\. M\. Ferreira, and A\. Bessani, “Processing Tweets for Cybersecurity Threat Awareness,”*Inf\. Syst\.*, vol\. 95, pp\. 101586, 2021, doi: 10\.1016/j\.is\.2020\.101586\.
- \[5\]B\.\-D\. Le, G\. Wang, M\. Nasim, and M\. Babar, “Gathering Cyber Threat Intelligence from Twitter Using Novelty Classification,” in*Proc\. 2019 Int\. Conf\. Cyberworlds \(CW\)*, pp\. 316–323, 2019, doi: 10\.1109/cw\.2019\.00058\.
- \[6\]A\. Rodriguez and K\. Okamura, “Generating Real Time Cyber Situational Awareness Information Through Social Media Data Mining,” in*Proc\. IEEE 43rd Annu\. Comput\. Softw\. Appl\. Conf\. \(COMPSAC\)*, vol\. 2, pp\. 502–507, 2019, doi: 10\.1109/compsac\.2019\.10256\.
- \[7\]O\. Alsodi, X\. Zhou, R\. Gururajan, A\. Shrestha, and E\. Btoush, “From Tweets to Threats: A Survey of Cybersecurity Threat Detection Challenges, AI\-Based Solutions and Potential Opportunities in X,”*Appl\. Sci\.*, vol\. 15, no\. 7, pp\. 3898, 2025, doi: 10\.3390/app15073898\.
- \[8\]H\.\-S\. Shin, H\.\-Y\. Kwon, and S\.\-J\. Ryu, “A New Text Classification Model Based on Contrastive Word Embedding for Detecting Cybersecurity Intelligence in Twitter,”*Electronics*, vol\. 9, no\. 9, pp\. 1527, 2020, doi: 10\.3390/electronics9091527\.
- \[9\]Y\. Fang, J\. Gao, Z\. Liu, and C\. Huang, “Detecting Cyber Threat Event from Twitter Using IDCNN and BiLSTM,”*Appl\. Sci\.*, vol\. 10, no\. 17, pp\. 5922, 2020, doi: 10\.3390/app10175922\.
- \[10\]S\. A\. Murad, A\. Dahal, and N\. Rahimi, “Multilingual Cyber Threat Detection in Tweets/X Using ML, DL, and LLM: A Comparative Analysis,”*IEEE Trans\. Comput\. Soc\. Syst\.*, vol\. 13, no\. 2, pp\. 1758–1772, 2025, doi: 10\.1109/tcss\.2025\.3623021\.
- \[11\]Y\. Park and W\. You, “A Pretrained Language Model for Cyber Threat Intelligence,” in*Proc\. 2023 Conf\. Empir\. Methods Nat\. Lang\. Process\. \(EMNLP\): Ind\. Track*, pp\. 113–122, 2023, doi: 10\.18653/v1/2023\.emnlp\-industry\.12\.
- \[12\]N\. Pangakis and S\. Wolken, “Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM\-Generated Training Labels,” in*Proc\. 6th Workshop Nat\. Lang\. Process\. Comput\. Soc\. Sci\. \(NLP\+CSS\)*, pp\. 113–131, 2024, doi: 10\.18653/v1/2024\.nlpcss\-1\.9\.
- \[13\]R\. Marinho and R\. Holanda, “Automated Emerging Cyber Threat Identification and Profiling Based on Natural Language Processing,”*IEEE Access*, vol\. 11, pp\. 58915–58936, 2023, doi: 10\.1109/access\.2023\.3260020\.
- \[14\]M\. Bayer, T\. Frey, and C\. Reuter, “Multi\-Level Fine\-Tuning, Data Augmentation, and Few\-Shot Learning for Specialized Cyber Threat Intelligence,”*Comput\. & Secur\.*, vol\. 134, pp\. 103430, 2023, doi: 10\.1016/j\.cose\.2023\.103430\.
- \[15\]B\. N\. Kebeli and H\.\-Y\. Kwon, “CTI\-ANN: Self\-Training\-Based Annotation With Tailored Augmentation for Cyber Threat Intelligence Posts,”*IEEE Trans\. Ind\. Inform\.*, vol\. 22, no\. 3, pp\. 1996–2007, 2026, doi: 10\.1109/TII\.2025\.3633628\.
- \[16\]Y\. You, Z\. Jiang, K\. Zhang, J\. Jiang, X\. Wang, Z\. Zhang, S\. Wang, and H\. Feng, “TI\-Prompt: Towards a Prompt Tuning Method for Few\-shot Threat Intelligence Twitter Classification,” in*Proc\. IEEE 46th Annu\. Comput\. Softw\. Appl\. Conf\. \(COMPSAC\)*, pp\. 272–279, 2022, doi: 10\.1109/COMPSAC54236\.2022\.00046\.
- \[17\]F\. Gilardi, M\. Alizadeh, and M\. Kubli, “ChatGPT Outperforms Crowd\-Workers for Text\-Annotation Tasks,”*Proc\. Natl\. Acad\. Sci\.*, vol\. 120, no\. 30, pp\. e2305016120, 2023, doi: 10\.1073/pnas\.2305016120\.
- \[18\]X\. He, Z\. Lin, Y\. Gong, A\.\-L\. Jin, H\. Zhang, C\. Lin, J\. Jiao, S\. M\. Yiu, N\. Duan, and W\. Chen, “AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators,” in*Proc\. 2024 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\.: Hum\. Lang\. Technol\. \(NAACL\-HLT\)*, pp\. 165–190, 2024, doi: 10\.18653/v1/2024\.naacl\-industry\.15\.
- \[19\]P\. Bansal and A\. Sharma, “Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost,”*arXiv preprint arXiv:2306\.15766*, 2023, doi: 10\.48550/arXiv\.2306\.15766\.
- \[20\]H\. Kim, K\. Mitra, R\. L\. Chen, S\. Rahman, and D\. Zhang, “MEGAnno\+: A Human\-LLM Collaborative Annotation System,” in*Proc\. 18th Conf\. Eur\. Chapter Assoc\. Comput\. Linguist\. \(EACL\): Syst\. Demonstr\.*, pp\. 168–176, 2024, doi: 10\.18653/v1/2024\.eacl\-demo\.18\.
- \[21\]X\. Wang et al\., “Human\-LLM Collaborative Annotation Through Effective Verification of LLM Labels,” in*Proc\. 2024 CHI Conf\. Hum\. Factors Comput\. Syst\.*, pp\.1–21, 2024, doi: 10\.1145/3613904\.3641960\.
- \[22\]T\. Kuzman and N\. Ljubešić, “LLM Teacher\-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification,”*IEEE Access*, vol\. 13, pp\.35621–35633, 2025, doi: 10\.1109/ACCESS\.2025\.3544814\.
- \[23\]V\. Behzadan, C\. Aguirre, A\. Bose, and W\. Hsu, “Corpus and Deep Learning Classifier for Collection of Cyber Threat Indicators in Twitter Stream,” in*Proc\. 2018 IEEE Int\. Conf\. Big Data*, pp\. 5002–5007, 2018, doi: 10\.1109/BigData\.2018\.8622506\.
- \[24\]H\. Lughbi, M\. Mars, and K\. Almotairi, “CybAttT: A Dataset of Cyberattack News Tweets for Enhanced Threat Intelligence,”*Data*, vol\. 9, no\. 3, pp\. 39, 2024, doi: 10\.3390/data9030039\.
- \[25\]M\. Bayer, P\. Kuehn, R\. Shanehsaz, and C\. Reuter, “CySecBERT: A Domain\-Adapted Language Model for the Cybersecurity Domain,”*ACM Trans\. Priv\. Secur\.*, vol\. 27, no\. 2, Art\. no\. 18, pp\. 1–20, 2024, doi: 10\.1145/3652594\.
- \[26\]E\. Aghaei, S\. Jain, P\. Arun, and A\. Sambamoorthy, “SecureBERT 2\.0: Advanced Language Model for Cybersecurity Intelligence,” in*Proc\. 2025 Annu\. Comput\. Secur\. Appl\. Conf\. Workshops \(ACSAC Workshops\)*, pp\. 382–388, 2025, doi: 10\.1109/ACSACW69556\.2025\.00071\.
- \[27\]Jackaduma, “SecRoBERTa,” Hugging Face Model Card, Available: https://huggingface\.co/jackaduma/SecRoBERTa