AK_QMUL 参加 ArGuard 共享任务：使用单一微调 JAIS 模型检测阿拉伯语有害提示词

# AK_QMUL at ArGuard Shared Tasks: A Single Fine-Tuned JAIS for Arabic Harmful-Prompt Detection
Source: [https://arxiv.org/html/2610.36360](https://arxiv.org/html/2610.36360)
Arkaitz ZubiagaAffiliation:Queen Mary University of LondonEmail:[a\.m\.khairallah@qmul\.ac\.uk](mailto:)a\.zubiaga@qmul\.ac\.uk

###### Abstract

We present AK\_QMUL’s submissions to the ArGuard\-2026 shared tasks on Arabic harmful\-prompt detection: a binary safe/unsafe task \(B1\) and a seven\-way harm\-domain task \(B2\)\. Both submissions use one backbone – JAIS\-6\.7B fine\-tuned with 4\-bit QLoRA and a linear classification head over its last\-token representation\. For B1 we train a four\-way head that factors the label by dialect register \{safe, unsafe\}×\\times\{MSA, other\}, we augment the training fold with MSA prompts from AraSafe, and decode with a plain argmax to score safe/unsafe\. For B2 we fine\-tune the same backbone as a seven\-way classifier and average the softmax outputs of five random seeds\. Both submissions ranked first on their track, with macro\-F1 0\.984 on B1 and 0\.790 on B2\.111Code and the train/development splits:[https://github\.com/alikhairallah/AK\_QMUL\-ArGuard\-2026](https://github.com/alikhairallah/AK_QMUL-ArGuard-2026)We report results by dialect and by class: on B1 the model is accurate across all six dialects \(93\.6–99\.8%\) and its predicted unsafe rate tracks the true rate closely; on B2 six of the seven classes reach F1 between 0\.73 and 0\.98, with the residual error concentrated in the*Bully*class, which overlaps semantically with*harassment*and*harm\-to\-others*\.

## 1Introduction

Large language models now are capable of following instructions in Arabic well enough to be useful, yet for the same reason, well enough to be misused[Khairallah and Zubiaga \(2025\)](https://arxiv.org/html/2610.36360#bib.bib4)\. A deployed assistant has to decide before it answers whether a prompt is asking for something harmful, and it has to make that decision across Modern Standard Arabic \(MSA\) and a wide range of dialects, where surface form varies far more than in English[Sengupta et al\. \(2023\)](https://arxiv.org/html/2610.36360#bib.bib6)\. The ArGuard\-2026 shared task[Alam et al\. \(2026\)](https://arxiv.org/html/2610.36360#bib.bib1)frames this as two problems over the same Arabic prompts: B1 asks whether a prompt is*safe*or*unsafe*, and B2 asks which of the seven harm domains an unsafe prompt belongs to\. Both are scored by macro\-F1, which weights the rare classes as heavily as the common ones and therefore rewards a detector that is uniform across categories rather than one that is accurate on average\.

Our approach uses one Arabic\-centric decoder for both tasks\. We fine\-tune JAIS\-6\.7B with 4\-bit QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.36360#bib.bib2);[Hu et al\., 2021](https://arxiv.org/html/2610.36360#bib.bib3)\), read a fixed\-size representation from its last token, and attach a linear classification head\. The backbone, the optimiser settings, and the pooling are shared; only the label set and two task\-specific configurations change\. For B1 we train a four\-way head that factors the label by dialect register and decodes it with a plain argmax; for B2 we average five seeds of the same seven\-way model\. Both submissions ranked first on their track, with macro\-F1 0\.984 on B1 and 0\.790 on B2\.

This paper describes the two systems and reports their results: \(i\) a single QLoRA\-tuned JAIS backbone serves both tasks; \(ii\) for B1, a dialect\-factored four\-way head with AraSafe augmentation and a plain\-argmax decode; \(iii\) for B2, a five\-seed softmax average; and \(iv\) a per\-dialect and per\-class breakdown showing where each system succeeds and where its residual error lies\.

## 2Task and Data

Both tasks share a pool of Arabic prompts[Zaghouani et al\. \(2026\)](https://arxiv.org/html/2610.36360#bib.bib7), each annotated with a dialect label: MSA or one of five regional dialects \(Syrian, Egyptian, Palestinian, Moroccan, and Algerian\), and a harm label\. B1 is a binary task over the full pool; B2 restricts attention to the unsafe prompts and assigns each to one of seven harm domains: adult\_content, harm\-to\-others, Bully, self\-harm, FraudandDeception, harassment, and hate\_speech\. We were provided with labelled train and development splits and an unlabelled test split\. The official metric on both tracks is macro\-F1\.

Two properties of the data shape the system\. First, the harm rate is not uniform across dialects \(Table[2](https://arxiv.org/html/2610.36360#S4.T2)\); MSA prompts are unsafe 37% of the time, while the five dialects range from 72% to 91%\. Second, the B2 classes are unbalanced;*adult\_content*is nearly four times as frequent as*hate\_speech*, and some pairs of classes describe overlapping behaviour, which is where the macro\-F1 penalty concentrates\.

## 3Our System

PromptJAIS\-6\.7BQLoRA4\-wayheadunsafe\_msaunsafe\_othersafe\_msasafe\_otherunsafesafeargmax

Figure 1:Task B1 system: JAIS with a four\-way \{safe, unsafe\}×\\times\{MSA, other\} head\. Colour marks the safe/unsafe axis and the\_msa/\_othersuffix the register axis; the argmax over the four logits collapses by prefix onto the binary*safe*/*unsafe*label\.Our two submissions share the exact backbone and the training recipe; they are otherwise different systems and differ in the classification head, in whether external data augments training and in how the final predictions are formed \(Table[1](https://arxiv.org/html/2610.36360#S3.T1)\)\. The following subsections will describe the shared recipe first, then each task’s system in turn\.

Table 1:Configuration of the two submitted systems\. Both share the JAIS\-6\.7B QLoRA backbone, the last\-token linear head, and the training recipe of Section[3\.1](https://arxiv.org/html/2610.36360#S3.SS1); they differ in the rows above\.### 3\.1Shared backbone and training

We use JAIS\-6\.7B[Sengupta et al\. \(2023\)](https://arxiv.org/html/2610.36360#bib.bib6), an Arabic\-centric decoder, loaded in 4\-bit NF4 with double quantisation and abfloat16compute dtype\. We attach LoRA adapters[Hu et al\. \(2021\)](https://arxiv.org/html/2610.36360#bib.bib3)to all linear projections \(r=16,α\\alpha=32, dropout 0\.05\) and keep the base weights frozen\. Because JAIS is a causal model, we pool by taking the hidden state at the last non\-padding position of the top layer and pass it infloat32through a single linear classification head\.

We fine\-tune for three epochs with AdamW at a learning rate of10−410^\{\-4\}, a linear schedule with 5% warmup, batch size 8, and maximum sequence length of 192 tokens, which covers the prompts without truncation in nearly all cases\. The loss is class\-weighted cross\-entropy with weights set to inverse class frequency, so the rare classes are not drowned out, which directly matters for a macro\-F1 objective\. We build training folds with stratified groupkk\-fold \(k=7k\{=\}7\)\. For the final test submission we refit on the concatenation of the train and development splits\. Everything above is identical for the two tasks; the differences below are what make them distinct systems\.

### 3\.2Task B1: a dialect\-factored head

We split each B1 label by dialect register into four classes: safe\-MSA, safe\-other, unsafe\-MSA, and unsafe\-other, and we train the head to predict all four\. The four\-way factoring lets the model represent the safe/unsafe boundary separately for standard and dialectal Arabic\. At inference we take the plain argmax over the four logits and collapse it to the binary decision: both*unsafe*sub\-classes map to*unsafe*\(Figure[1](https://arxiv.org/html/2610.36360#S3.F1)\)\. The decode uses no dialect\-based post\-processing; the dialect information enters only through the labels the model is trained on\.

Under this four\-way scheme, the B1 training pool \(train\+dev, 19,236 prompts\) is dominated by the two unsafe classes, unsafe\-MSA \(40\.6%\) and unsafe\-other \(39\.3%\), while the two safe classes are much smaller: safe\-other \(12\.5%\) and safe\-MSA \(7\.6%\)\. As safe\-MSA is thus the least represented, and to strengthen its coverage, we augment the training fold with safe, human\-written MSA prompts from AraSafe[Mubarak et al\. \(2025\)](https://arxiv.org/html/2610.36360#bib.bib5), the shared task’s named example of permitted external data\. From the AraSafe human\-written set \(12,077 prompts\) we keep non\-harmful instances, drop any shorter than ten characters or matching a dialectal\-marker lexicon, and deduplicate against the train, development, and test prompts\. This yields 7,607 prompts, all in Modern Standard Arabic, all safe, and added to the training fold only \(labelled safe\-MSA\), while the held\-out fold is left untouched and the augmentation is applied solely for the final test refit\.

### 3\.3Task B2: a seed\-averaged classifier

Promptseed 42seed 43seed 44seed 45seed 46averagesoftmax1 of 7classesargmax5 seeds, each 7\-way softmax

Figure 2:Task B2 system: five seeds \(42–46\) of the same seven\-way JAIS classifier run in parallel; their softmax distributions are averaged and the argmax of the mean is the prediction\.B2 uses the same backbone with a seven\-way head and no dialect factoring or augmentation\. We train five identically configured seeds \(42–46\), average their softmax distributions, and take the argmax of the mean as the prediction \(Figure[2](https://arxiv.org/html/2610.36360#S3.F2)\)\. Averaging is the only ensembling step; there is no stacking, no held\-out meta\-learner, and no per\-class thresholding\.

## 4Results

Table 2:B1 test set by dialect: number of prompts, true unsafe rate, our model’s predicted unsafe rate, and accuracy\.Table 3:B1 ablation on the official test set\. Rows 1–2 remove one design choice at a time from the submitted system \(row 3\): the dialect\-register factoring and the AraSafe augmentation\. All rows share the same JAIS\-6\.7B QLoRA backbone, seed, held\-out fold, and training recipe; only the ablated factor changes\.Table 4:B2 per\-class F1 by dialect on the test set \(dash = no test examples of that class in the dialect\)\. Per\-dialect macro\-F1 is over the classes present in that dialect;nnis the number of unsafe prompts per dialect\. The*Bully*weakness is confined to MSA\.Both submissions ranked first on their track\. B1 reaches macro\-F1 0\.984, and its predicted unsafe rate \(3888 of 7809 prompts\) is within 1% of the true rate \(3851\)\. B2 reaches 0\.790; the five seeds individually scored between 0\.779 and 0\.785 on the test set, and averaging their softmax outputs before the argmax gives 0\.790\.

We next break down the results by different dimensions –dialect and class– to further delve into the analysis of results\.

#### B1 by dialect\.

Table[2](https://arxiv.org/html/2610.36360#S4.T2)breaks B1 down by dialect\. Accuracy is high in every dialect, from 93\.6% \(Moroccan\) to 99\.8% \(Egyptian\), and the model’s predicted unsafe rate stays close to the true rate, with 37\.1% against 36\.6% for MSA, 91\.6% against 91\.3% for Egyptian\. The model uses dialect as evidence, through the labels it was trained on, without turning it into a fixed rule: dialectal prompts that are in fact safe are still classified as safe\.

#### B1 ablation\.

Table[3](https://arxiv.org/html/2610.36360#S4.T3)isolates the two B1 design choices, each removed at a time while everything else is held fixed\. The AraSafe augmentation is the most effective component; without it, the system scores only 0\.624\. The training pool is about 80% unsafe, whereas the blind test is close to balanced, so the un\-augmented model carries the training prior and flags most prompts as unsafe \(6,426 of 7,809 predicted unsafe against 3,851 truly unsafe\)\. The four\-way register head adds a small margin, from 0\.983 to 0\.984, concentrated in the Modern Standard Arabic safe/unsafe boundary rather than in the overall score\.

Table 5:Per\-class precision \(P\), recall \(R\), and F1 for B2 on the test set, sorted by F1\. The macro average is 0\.790\.
#### B2 by class\.

Table[5](https://arxiv.org/html/2610.36360#S4.T5)breaks B2 down by class\. Six of the seven classes reach F1 between 0\.73 and 0\.98\. The residual error is concentrated in*Bully*: its precision is 0\.815 but its recall is only 0\.443, meaning the model finds fewer than half of the true*Bully*prompts and assigns most of the rest to*harassment*and*harm\-to\-others*\. This is not a matter of data shortage,*Bully*has 908 examples, more than four of the classes that score higher\. With performance scores significantly superior for the other classes, this suggests that further research in improving the model should focus on achieving competitive performance on the*Bully*class\.

#### B2 by class and dialect\.

Table[4](https://arxiv.org/html/2610.36360#S4.T4)crosses the two breakdowns, and it reframes the*Bully*result\. Two patterns emerge\. First,*adult\_content*is uniformly easy \(F1 0\.95–1\.00 in every dialect\), so its high score is not a register effect\. Second, the*Bully*weakness is not a property of the class but of Modern Standard Arabic: Bully reaches 0\.81–0\.97 in all five dialects, and its low aggregate \(0\.57\) comes almost entirely from MSA \(0\.46\), where 719 of its 908 test prompts fall\. The other harm classes, like*harm\-to\-others*,*self\-harm*, and*FraudandDeception*, show the same shape, dropping in MSA and recovering in other dialects\. MSA is consequently the hardest register \(macro\-F1 0\.70\), while every dialect scores 0\.88–0\.95\. The confusion between neighbouring harm domains is therefore an MSA phenomenon rather than a dialectal one; the non\-MSA per\-class cells rest on 14–239 examples and should be read as indicative\.

## 5Conclusion

This paper describes our participation in the ArGuard shared task as part of ArabicNLP\. A single Arabic decoder, JAIS\-6\.7B fine\-tuned with QLoRA and read out with a linear head, ranked first on both ArGuard\-2026 tracks: macro\-F1 0\.984 on B1 and 0\.790 on B2\. B1 uses a four\-way head that factors the label by dialect register, AraSafe augmentation to cover the least present class, and a plain\-argmax decode with no dialect routing; the resulting system is accurate across all six dialects\. B2 uses the same backbone as a seven\-way classifier with the softmax outputs of five seeds averaged; its residual error sits almost entirely in the*Bully*class and within that class in Modern Standard Arabic, where it is confused with two neighbouring categories\.

## Limitations

Our system relies on a single Arabic\-centric backbone with a linear head; we do not compare alternative Arabic encoders or architectures, so the results speak to this recipe rather than to the best attainable on this task\. The B1 ablation isolates the two design choices on the official test split under the submitted seed and fold; we do not repeat it across seeds\. The main residual error, the*Bully*class within Modern Standard Arabic, remains unresolved: it is confused with*harassment*and*harm\-to\-others*domains, and closing that gap likely needs finer annotation rather than additional data\.

## Ethics Statement

This work targets the detection of harmful Arabic prompts, a defensive application intended to support content moderation and safer deployment of Arabic language models\. We work only with the data released for the ArGuard\-2026 shared task and with AraSafe, both used under permitted terms, and we release no new harmful content\. Our repository contains the system code and the train/development splits only\.

## References

- Alam et al\. \(2026\)Firoj Alam, Md\. Rafiul Biswas, Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Hamdy Mubarak, Georgios Mikros, Abul Hasnat, and Wajdi Zaghouani\. 2026\.ArGuard Shared Task: Harmful content detection in Arabic Memes and LLM Prompts\.In*Proceedings of the Fourth Arabic Natural Language Processing Conference: Shared Tasks*, Budapest, Hungary\. Association for Computational Linguistics\.
- Dettmers et al\. \(2023\)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\. 2023\.Qlora: Efficient finetuning of quantized llms\.*Advances in neural information processing systems*, 36:10088–10115\.
- Hu et al\. \(2021\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2021\.Lora: Low\-rank adaptation of large language models\.*arXiv preprint arXiv:2106\.09685*\.
- Khairallah and Zubiaga \(2025\)Ali Khairallah and Arkaitz Zubiaga\. 2025\.Alhd: A large\-scale and multigenre benchmark dataset for arabic llm\-generated text detection\.*arXiv preprint arXiv:2510\.03502*\.
- Mubarak et al\. \(2025\)Hamdy Mubarak, Abubakr Mohamed, and Majd Hawasly\. 2025\.[AraSafe: Benchmarking safety in Arabic LLMs](https://doi.org/10.18653/v1/2025.findings-emnlp.529)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 9976–9992, Suzhou, China\. Association for Computational Linguistics\.
- Sengupta et al\. \(2023\)Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al\. 2023\.Jais and jais\-chat: Arabic\-centric foundation and instruction\-tuned open generative large language models\.*arXiv preprint arXiv:2308\.16149*\.
- Zaghouani et al\. \(2026\)Wajdi Zaghouani, Md\. Rafiul Biswas, Kholoud Khalil Aldous, and Mabrouka Bessghaier\. 2026\.[Arabicdialectsafety: A dialect\-aware benchmark for arabic content safety classification](https://doi.org/10.48550/arXiv.2608.01291)\.*arXiv preprint arXiv:2608\.01291*\.