HalluScore: Large Language Model Hallucination Question Answering Benchmark

# HalluScore: Large Language Model Hallucination Question Answering Benchmark
Source: [https://arxiv.org/html/2605.17007](https://arxiv.org/html/2605.17007)
\\pagemark

1\\addauthorAisha Alansari10000\-0002\-4600\-976X\\addauthor\[corresponding\]Hamzah Luqman1,20000\-0001\-7944\-5093\\addaffiliation1Department of Information and Computer Science, King Fahd University of Petroleum and Minerals\\addaffiliation2SDAIA\-KFUPM Joint Research Center for Artificial Intelligence\\correspondingauthorhluqman@kfupm\.edu\.sa\\setabstractLarge language models \(LLMs\) have achieved remarkable progress in natural language generation, but remain susceptible to hallucination\. In response to growing concerns about hallucinations, several benchmarks have been developed, primarily in English and Chinese\. However, Arabic remains underrepresented, with limited benchmarks for LLMs hallucination due to scarce annotated resources and the language’s morphological complexity\. Consequently, existing benchmarks do not adequately reflect the linguistic, cultural, and reasoning characteristics of Arabic\. To address this gap, we introduceHalluScore, a structured Arabic question answering benchmark designed to evaluate hallucination behavior in LLMs across different levels of reasoning difficulty, various knowledge domains, historical timelines, and culturally grounded Arabic scenarios\. It contains 827 carefully curated questions for evaluating, detecting, and mitigating hallucination in LLMs\. The dataset was constructed through a structured pipeline involving quality assurance, filtering for clarity and factual validity, and model\-driven selection to retain questions that consistently trigger hallucinations\. Each question is linked to verified ground\-truth evidence, answer explanations, and multi\-label annotations\. Using theHalluScorebenchmark, we conduct a comprehensive empirical analysis of hallucination patterns across 17 Arabic, multilingual, and reasoning LLMs\. Moreover, we provide high\-quality human annotations identifying hallucinated, non\-hallucinated, and partially hallucinated responses of all evaluated LLMs\. These results suggest that hallucination in Arabic LLMs extends beyond factual inaccuracies, encompassing challenges related to cultural understanding, linguistic reasoning, and logical consistency\. We release HalluScore to support future research on improving the reliability and cultural competence of LLMs in Arabic\.

###### keywords:

Large Language Models; LLMs; Hallucination; Hallucination Benchmark; Hallucination Evaluation; Question Answering

## 1Introduction

Recent large language models \(LLMs\), such as GPT\-5 and Claude\-4, have been developed as open\-domain chatbots capable of answering questions across a wide range of topics\. Notwithstanding their remarkable performance, these LLMs sometimeshallucinate, generating plausible; yet, non\-factual responses\[[1](https://arxiv.org/html/2605.17007#bib.bib1)\]\. Hallucination in LLMs refers to the phenomenon in which models generate outputs that are not grounded in verified facts or reliable sources\[[1](https://arxiv.org/html/2605.17007#bib.bib1)\]\. It occurs when an LLM generates responses that sound fluent and convincing but contain inaccurate, misleading, or completely fabricated information\. LLM hallucination is commonly categorized as factual and faithfulness hallucinations\[[2](https://arxiv.org/html/2605.17007#bib.bib2)\]\. In factual hallucination, an LLM produces incorrect information that contradicts verifiable knowledge, whereas in faithfulness hallucinations, an LLM generates content that is not supported by the provided input or context, even if it is factually correct\[[1](https://arxiv.org/html/2605.17007#bib.bib1)\]\. Hallucination can emerge at multiple stages of the LLM development pipeline, including the data collection stage, due to outdated or knowledge\-conflicting data, during the fine\-tuning stage, due to task\-specific biases and misalignment between the LLM’s internal capabilities and the expectations encoded in the alignment data, or during the inference stage, due to sampling randomness and softmax activation\[[2](https://arxiv.org/html/2605.17007#bib.bib2),[3](https://arxiv.org/html/2605.17007#bib.bib3)\]\. Addressing hallucinations in LLMs is crucial to improving their reliability and trustworthiness in real\-world applications, such as healthcare, education, and law, where incorrect information can lead to harmful decisions\.

In response to growing concerns about hallucinations in LLMs, several benchmarks have been proposed to evaluate their factual reliability\[[4](https://arxiv.org/html/2605.17007#bib.bib4),[5](https://arxiv.org/html/2605.17007#bib.bib5)\]\. However, most of these datasets target high\-resource languages, such as English and Chinese, leaving widely spoken languages, such as Arabic, comparatively underexplored\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\. This gap persists despite the recent rise of Arabic LLMs, such as Allam\[[7](https://arxiv.org/html/2605.17007#bib.bib7)\], Jais\[[8](https://arxiv.org/html/2605.17007#bib.bib8)\], and Fanar\[[9](https://arxiv.org/html/2605.17007#bib.bib9)\], which have attracted increasing attention from both academia and industry\[[10](https://arxiv.org/html/2605.17007#bib.bib10),[11](https://arxiv.org/html/2605.17007#bib.bib11)\]\. Nevertheless, hallucination in Arabic LLMs remains underexplored, particularly from the perspectives of systematic evaluation, detection, and mitigation\[[3](https://arxiv.org/html/2605.17007#bib.bib3)\]\.

This research gap is further amplified by the linguistic properties of Arabic itself\. The rich morphology and complex syntactic structure of Arabic pose additional challenges for natural language understanding and generation systems\[[12](https://arxiv.org/html/2605.17007#bib.bib12),[13](https://arxiv.org/html/2605.17007#bib.bib13)\]\. These characteristics increase the level of ambiguity and variability in model outputs, making hallucination detection and mitigation more challenging compared to structurally simpler languages\. As Arabic LLMs continue to be integrated into real\-world applications, ensuring their factual reliability becomes increasingly critical\. Therefore, developing dedicated Arabic hallucination benchmarks and conducting systematic evaluations of LLMs’ hallucination is both timely and necessary\.

Few studies have recently been proposed to evaluate and detect LLM hallucinations in the Arabic context\[[14](https://arxiv.org/html/2605.17007#bib.bib14),[15](https://arxiv.org/html/2605.17007#bib.bib15),[3](https://arxiv.org/html/2605.17007#bib.bib3),[16](https://arxiv.org/html/2605.17007#bib.bib16),[17](https://arxiv.org/html/2605.17007#bib.bib17)\]\. Although Halwasa\[[14](https://arxiv.org/html/2605.17007#bib.bib14)\]represents the first dataset developed for Arabic hallucination detection and mitigation, it mainly focuses on text generation conditioned on predefined keywords, which may not fully reflect realistic user interactions or complex reasoning scenarios\. Similarly, other datasets, such as Aftina\[[16](https://arxiv.org/html/2605.17007#bib.bib16)\]and IslamicEval\[[17](https://arxiv.org/html/2605.17007#bib.bib17)\], are limited to specific domains, particularly the religious domain, which restricts their generalizability to broader real\-world applications\. These limitations highlight the need for a more comprehensive, multi\-domain Arabic question answering \(QA\) benchmark that can better capture the diverse conditions under which hallucinations occur, including adversarial phrasing, cultural knowledge, and reasoning complexity\.

To address these gaps, we introduceHalluScore, a structured Arabic QA benchmark comprising 827 QA pairs designed to systematically assess LLM hallucinations across multiple dimensions, including domain knowledge, levels of reasoning, historical events, cultural knowledge, and adversarial question types\. The proposed dataset is constructed through a multi\-stage, structured pipeline involving question collection, quality filtering, hallucination\-driven selection, and manual refinement to ensure diversity, clarity, and hallucination relevance\. Each question is associated with a verified ground\-truth source, an answer explanation, and an annotation with multiple labels, including the type of question, domain knowledge, and binary indicators, including reasoning requirement, adversarial intent, cultural relevance in Arabic, and historical dependency\. We used this dataset to evaluate 17 LLMs, and their responses have been carefully annotated by humans as hallucinations or non\-hallucinations following clearly defined criteria\. We also evaluate partial hallucination in non\-hallucination responses that address the main question but introduce additional fabricated facts\. Unlike many existing benchmarks that rely solely on automatic labeling or weak supervision, our annotations provide high\-quality ground truth for hallucination evaluation\. Additionally, we identify the types of questions that trigger hallucinations in each evaluated LLM, which is important for understanding model weaknesses and revealing whether certain hallucination risks are model\-specific or consistent across architectures\. The main contributions of this study can be summarized as follows:

- •IntroduceHalluScore, the first Arabic QA hallucination benchmark for evaluating hallucination in LLMs\.
- •Propose a novel multi\-dimensional hallucination category that captures hallucination\-related factors beyond binary correctness, including hallucination types, adversarial intent, reasoning requirements, historical relevance, Arabic cultural grounding, and domain\-specific knowledge\.
- •Provide verified ground\-truth evidence and detailed answer explanations for each sample to enable explainable evaluation, supporting LLM\-as\-a\-judge frameworks, and facilitating future research hallucination evaluation, detection, and mitigation\.
- •Benchmark 17 LLMs, including Arabic, multilingual, and reasoning\-based LLMs onHalluScore, and provide a detailed analysis of their responses\.
- •Conduct a human annotation of the 17 LLMs’ responses to categorize faithful and factual hallucination, as well as partial hallucination\.
- •Analyze the dominant hallucination types exhibited by different LLMs and identify which categories occur most frequently in each model\.
- •Discuss the weaknesses of LLMs through response\-level analysis to highlight failure cases related to cultural understanding, prompt sensitivity, and reasoning limitations\.

The remainder of the paper is organized as follows: Section 2 reviews related studies\. Then, a detailed discussion about the construction ofHalluScoreis presented in Section 3\. Section 4 presents a statistical analysis of the dataset and Section 5 details the benchmarking methodology by outlining the evaluated models, presenting the experimental setup, and explaining the human evaluation protocol\. The empirical results are discussed thoroughly in Section 6 with a detailed hallucination response\-level analysis\. Finally, Section 7 discusses the limitations, and Section 8 concludes this study\.

## 2Related work

#### LLMs hallucination\.

Hallucination in LLMs refers to the generation of content that lacks grounding in factual or accurate information\[[1](https://arxiv.org/html/2605.17007#bib.bib1),[4](https://arxiv.org/html/2605.17007#bib.bib4),[2](https://arxiv.org/html/2605.17007#bib.bib2)\]\. It occurs when the LLM tends to generate text that includes fictional, misleading, or entirely fabricated information\. This behavior is attributed to several reasons, such as outdated knowledge, the Softmax function, the attention mechanism, and sampling randomness\[[2](https://arxiv.org/html/2605.17007#bib.bib2)\]\. This issue undermines the trustworthiness of LLMs and limits their practical use in real\-world scenarios\. Therefore, addressing hallucinations in LLMs is crucial to improving their trustworthiness in real\-world applications such as finance, healthcare, and law\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\.

A growing body of work has focused on benchmarking hallucination behavior across languages and tasks\. Early studies primarily evaluated hallucinations in English settings, particularly in summarization and QA\[[1](https://arxiv.org/html/2605.17007#bib.bib1),[18](https://arxiv.org/html/2605.17007#bib.bib18),[19](https://arxiv.org/html/2605.17007#bib.bib19)\]\. More recent efforts have extended these analyses to multilingual contexts, including Chinese\[[20](https://arxiv.org/html/2605.17007#bib.bib20),[21](https://arxiv.org/html/2605.17007#bib.bib21),[22](https://arxiv.org/html/2605.17007#bib.bib22)\]and Arabic\[[14](https://arxiv.org/html/2605.17007#bib.bib14),[3](https://arxiv.org/html/2605.17007#bib.bib3),[17](https://arxiv.org/html/2605.17007#bib.bib17)\], reflecting an increasing interest in understanding hallucination beyond high\-resource languages\. However, hallucination remains challenging in low\-resource languages due to linguistic complexity, limited data availability, and domain\-specific knowledge gaps\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\.

To detect hallucination in LLMs, previous research has proposed various detection strategies, which can be broadly categorized into retrieval\-, uncertainty\-, embedding\-, learning\- and self\-consistency\-based approaches\[[2](https://arxiv.org/html/2605.17007#bib.bib2),[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\. Retrieval\-based methods verify model outputs against external knowledge sources to compare generated content against supporting documents\[[23](https://arxiv.org/html/2605.17007#bib.bib23),[24](https://arxiv.org/html/2605.17007#bib.bib24)\]\. These techniques are effective for factual hallucination, but depend heavily on the quality and coverage of retrieved knowledge\. In contrast, uncertainty\-based approaches rely on model confidence signals, such as token probabilities and entropy, to flag unreliable outputs\[[25](https://arxiv.org/html/2605.17007#bib.bib25),[26](https://arxiv.org/html/2605.17007#bib.bib26)\]\. Although these methods are data\-efficient, they often fail when models generate hallucinated responses with high confidence\. Embedding\-based methods measure semantic consistency between inputs, outputs, and references, capturing deeper semantic discrepancies but struggling in out\-of\-domain scenarios\[[27](https://arxiv.org/html/2605.17007#bib.bib27),[28](https://arxiv.org/html/2605.17007#bib.bib28)\]\. Learning\-based approaches train classifiers on annotated data or internal representations to detect hallucinations, achieving strong performance but requiring high\-quality labeled datasets\[[29](https://arxiv.org/html/2605.17007#bib.bib29),[30](https://arxiv.org/html/2605.17007#bib.bib30)\]\. Finally, self\-consistency methods generate multiple outputs and assess their agreement, enabling detection without external knowledge, but remaining sensitive to prompt design and sampling strategies\[[31](https://arxiv.org/html/2605.17007#bib.bib31),[32](https://arxiv.org/html/2605.17007#bib.bib32)\]\.

On the other hand, to mitigate hallucination, the approaches can be grouped into prompt\-, retrieval\-, reasoning\-based, and model\-centric techniques\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\. Prompt\-based methods guide the model toward factual outputs through structured instructions\[[33](https://arxiv.org/html/2605.17007#bib.bib33),[34](https://arxiv.org/html/2605.17007#bib.bib34)\], while retrieval\-based approaches, such as retrieval\-augmented generation \(RAG\)\[[35](https://arxiv.org/html/2605.17007#bib.bib35)\], ground responses in external knowledge\. Reasoning\-based techniques, including chain\-of\-thought\[[36](https://arxiv.org/html/2605.17007#bib.bib36)\]and self\-verification\[[37](https://arxiv.org/html/2605.17007#bib.bib37)\], improve logical consistency and reduce reasoning errors\. Finally, model\-centric approaches focus on improving the model itself and reducing hallucination through fine\-tuning\[[38](https://arxiv.org/html/2605.17007#bib.bib38)\]or architectural modifications\[[39](https://arxiv.org/html/2605.17007#bib.bib39)\]\.

Despite the advancement in detecting and mitigating LLM hallucination, these techniques remain understudied in low\-resource languages such as Arabic\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\. Given the morphological complexity and dialectal variation of Arabic, dedicated benchmarks are essential to design hallucination detection and mitigation techniques tailored for the Arabic language\[[14](https://arxiv.org/html/2605.17007#bib.bib14),[17](https://arxiv.org/html/2605.17007#bib.bib17),[15](https://arxiv.org/html/2605.17007#bib.bib15)\]\.

#### Hallucination evaluation metrics\.

Evaluating hallucination typically involves either automatic or human\-based factuality measures\. Automatic metrics include factual consistency scores \(e\.g\., FEQA\[[40](https://arxiv.org/html/2605.17007#bib.bib40)\]\), entailment\-based metrics \(e\.g\., FactCC\[[41](https://arxiv.org/html/2605.17007#bib.bib41)\]\), and information overlap metrics \(e\.g\., FActScore\[[42](https://arxiv.org/html/2605.17007#bib.bib42)\]\)\. Human evaluation remains the gold standard, often providing fine\-grained labels such as factual, partially factual, or hallucinated\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\. Despite progress in automatic evaluation, human\-curated annotations continue to yield higher reliability, particularly in low\-resource and morphologically complex languages\[[43](https://arxiv.org/html/2605.17007#bib.bib43)\]\.

#### Hallucination benchmarks\.

A substantial body of research has focused on curating datasets to detect and mitigate hallucination across diverse Natural Language Generation \(NLG\) tasks\. These datasets fall into two categories: detection and mitigation datasets\[[5](https://arxiv.org/html/2605.17007#bib.bib5)\]\. Typically, hallucination detection datasets pair model inputs and outputs with explicit hallucination annotations, whereas mitigation datasets include contextually grounded references that guide model factuality\[[6](https://arxiv.org/html/2605.17007#bib.bib6)\]\.

Among existing datasets, QA and summarization are the primary NLG tasks used to evaluate hallucination, particularly in English and Chinese\[[4](https://arxiv.org/html/2605.17007#bib.bib4)\]\. Widely used English benchmarks include TruthfulQA\[[44](https://arxiv.org/html/2605.17007#bib.bib44)\], FreshQA\[[45](https://arxiv.org/html/2605.17007#bib.bib45)\], and WikiFact\[[46](https://arxiv.org/html/2605.17007#bib.bib46)\]\. Moreover, Chinese benchmarks for hallucination in LLMs include HalluQA\[[20](https://arxiv.org/html/2605.17007#bib.bib20)\]and UhgEval\[[21](https://arxiv.org/html/2605.17007#bib.bib21)\]\. In contrast, research on Arabic hallucination remains limited, with Halwasa\[[14](https://arxiv.org/html/2605.17007#bib.bib14)\]being one of the earliest datasets designed for Arabic text\-generation hallucination detection and mitigation\. However, this dataset primarily focuses on text generation guided by predefined keywords, which may not accurately represent real\-world user queries\. Other Arabic hallucination datasets are limited to one domain, such as religion\[[17](https://arxiv.org/html/2605.17007#bib.bib17),[16](https://arxiv.org/html/2605.17007#bib.bib16),[47](https://arxiv.org/html/2605.17007#bib.bib47),[48](https://arxiv.org/html/2605.17007#bib.bib48)\]\. IslamicEval\[[17](https://arxiv.org/html/2605.17007#bib.bib17)\]focuses on hallucination detection and mitigation in Islamic texts, particularly Qur’anic verses and Hadith quotations, whileAftina\[[16](https://arxiv.org/html/2605.17007#bib.bib16)\]provides a domain\-specific QA benchmark designed to support hallucination mitigation through referenced answers\. Other datasets focus only on evaluating hallucination in LLMs, includingAraHalluEval\[[3](https://arxiv.org/html/2605.17007#bib.bib3)\]\. This dataset introduces an Arabic hallucination evaluation setting that covers open\-ended QA and summarization tasks across multiple hallucination categories, including named\-entity and numerical errors, and supports training hallucination detection models using human\-annotated samples\.

Beyond Arabic\-only resources, several multilingual benchmarks are commonly used for cross\-lingual evaluation\. HalluVerse\[[15](https://arxiv.org/html/2605.17007#bib.bib15)\]supports multilingual hallucination type classification, enabling comparative analysis of hallucination patterns across languages\. Mu\-SHROOM\[[49](https://arxiv.org/html/2605.17007#bib.bib49)\]provides fine\-grained span\-level hallucination annotations useful for token\-level detection\. HalOmi\[[50](https://arxiv.org/html/2605.17007#bib.bib50)\]focuses on hallucination detection in machine translation with sentence\- and token\-level annotations across multiple languages\. Similarly, Poly\-FEVER\[[51](https://arxiv.org/html/2605.17007#bib.bib51)\]provides a multilingual fact verification benchmark that supports hallucination detection through claim verification tasks\. However, the coverage of Arabic samples remains limited in these datasets\.

To bridge the limitations of existing Arabic resources, we introduceHalluScore, the first Arabic hallucination benchmark specifically designed for generative QA\. The dataset consists of high\-quality human\-curated QA pairs spanning multiple question types, categories, and domains\. Among existing benchmarks, the closest datasets to our work are TruthfulQA\[[44](https://arxiv.org/html/2605.17007#bib.bib44)\], HalluQA\[[20](https://arxiv.org/html/2605.17007#bib.bib20)\], and HaluEval\[[18](https://arxiv.org/html/2605.17007#bib.bib18)\], as they focus on hallucination in generative QA\. Similar to these datasets,HalluScoretargets hallucination\-prone question types and factual reliability\. However, unlike prior work, our dataset is constructed entirely in Arabic and includes richer annotations that cover adversarial intent, reasoning requirements, cultural knowledge, and domain expertise, as summarized in Table[I](https://arxiv.org/html/2605.17007#S2.T1)\. Furthermore, with 827 QA pairs,HalluScoreprovides a larger evaluation set than the closest hallucination\-focused QA benchmarks, enabling more comprehensive and statistically reliable analysis of hallucination behavior across models\.

Table I:Comparison ofHalluScorewith existing QA hallucination benchmarks\.Dataset Features:F1 \(Multi\-domain\), F2 \(Manual collection\), F3 \(Human annotation\), F4 \(Ground\-truth evidence\), F5 \(Answer explanations\), F6 \(LLMs benchmarking for hallucination\)\.Question Types:F7 \(Pseudoscience questions\), F8 \(Reasoning\-based questions\), F9 \(Historical questions\), F10 \(Arabic culture\-oriented questions\), F11 \(Unanswerable questions\)\.DatasetLang\.\#QsTaskDataset FeaturesQuestion TypesF1F2F3F4F5F6F7F8F9F10F11HaluEval\[[18](https://arxiv.org/html/2605.17007#bib.bib18)\]English10KContext✓✗✓✓✗✓✗✗✓✗✗HotpotQA\[[52](https://arxiv.org/html/2605.17007#bib.bib52)\]English113KContext✓✓✗✓✗✗✗✓✓✗✗TriviaQA\[[53](https://arxiv.org/html/2605.17007#bib.bib53)\]English95KContext✓✓✗✓✗✗✗✓✓✗✗TruthfulQA\[[44](https://arxiv.org/html/2605.17007#bib.bib44)\]English817Generative✓✓✓✓✗✓✓✓✗✗✓FreshQA\[[45](https://arxiv.org/html/2605.17007#bib.bib45)\]English600Generative✓✓✗✓✗✓✗✓✓✗✓MedHallu\[[54](https://arxiv.org/html/2605.17007#bib.bib54)\]English10KGenerative✗✗✗✓✗✓✗✗✗✗✗DefAn\[[55](https://arxiv.org/html/2605.17007#bib.bib55)\]English75KGenerative✓✓✗✗✗✓✗✓✓✗✗HalluQA\[[20](https://arxiv.org/html/2605.17007#bib.bib20)\]Chinese450Generative✓✓✓✓✗✓✓✓✓✗✓IslamicEval\[[17](https://arxiv.org/html/2605.17007#bib.bib17)\]Arabic1\.5KGenerative✗✓✗✓✗✓✗✓✓✗✗AraHalluEval\[[3](https://arxiv.org/html/2605.17007#bib.bib3)\]Arabic200Generative✓✗✓✗✗✓✗✗✓✗✗HalluScoreArabic827Generative✓✓✓✓✓✓✓✓✓✓✓

## 3The HalluScore Benchmark

To develop a comprehensive benchmark for evaluating, detecting, and mitigating LLM hallucinations in Arabic content, we constructedHalluScoreusing a multi\-stage, human\-supervised pipeline\. The dataset is designed to evaluate hallucination behavior in LLMs across different levels of reasoning difficulty, various knowledge domains, historical timelines, and culturally grounded Arabic scenarios\. Figure[1](https://arxiv.org/html/2605.17007#S3.F1)illustrates the pipeline of theHalluScoredataset construction and benchmarking\. The data collection process involved writing QA pairs via crowdsourcing and translating a few QA pairs from TruthfulQA\[[44](https://arxiv.org/html/2605.17007#bib.bib44)\]\. All crowdsourced data were human\-curated by native Arabic speakers to ensure linguistic quality and cultural authenticity\. The collected QA pairs were then inspected and filtered based on several criteria to ensure data quality\. Furthermore, the questions were evaluated using multiple LLMs to identify those that are more likely to induce hallucinations, and only the most challenging QA pairs were retained\. We then categorized the QA pairs into 13 domain knowledge categories and further expanded the dataset by adding additional questions that were constructed without testing them on LLMs to avoid potential data leakage and evaluation bias\. The final version of the dataset consists of 827 samples, each question associated with a verified ground\-truth source and an explanation of the ground\-truth answer\. Each QA pair is annotated with multiple labels, including the type of question, domain knowledge, and binary indicators of reasoning, adversarial intent, cultural relevance in Arabic, and historical dependency\. We also provided a ground\-truth reference and an answer explanation to support hallucination mitigation research\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x1.png)Figure 1:The pipeline ofHalluScoredataset construction and benchmarking\.### 3\.1Initial Dataset Collection

In our previous work, AraHalluEval\[[3](https://arxiv.org/html/2605.17007#bib.bib3)\], which systematically evaluated Arabic, multilingual, and reasoning\-based LLMs in summarization and generative QA tasks, we observed that named entities and numbers were the categories that most frequently trigger hallucination in LLMs\. Furthermore, by translating TruthfulQA\[[44](https://arxiv.org/html/2605.17007#bib.bib44)\]into Arabic, we gained an initial understanding of the types of questions that lead to hallucination\. These findings motivated the development of a dedicated Arabic QA hallucination benchmark\.

We began collecting our dataset through crowdsourcing, where we defined 18 categories:Art, Economics, Education, Entertainment, Finance, Geography, Health, Language, Law, Mathematics, Nutrition, Politics, Psychology, Religion, Science, Sociology, Sports, and Technology\. The participants who collected the QA pairs were instructed to focus on questions that reflect common false beliefs \(imitative falsehood\), entities, numbers, rare knowledge \(long\-tail knowledge\), and unanswerable questions\. This instruction was given to ensure that the questions reflect cultural stereotypes, myths, conspiracies, superstitions, and false presuppositions prevalent in each category\. To further enrich the dataset, we also included a subset of questions translated from TruthfulQA\[[44](https://arxiv.org/html/2605.17007#bib.bib44)\], ensuring coverage of both Arabic\-related misconceptions and internationally relevant factual fallacies\. This process resulted in an initial pool of 1,500 QA pairs, each with a verified source link to support the ground\-truth answer\. The ground\-truth links were obtained from reliable sources, such as Wikipedia and official government websites\.

Figure 2:The domain knowledge categories included inHalluScorewith their definitions and examples\.![Refer to caption](https://arxiv.org/html/2605.17007v1/figures/Table_2.png)
### 3\.2Quality Assurance and Filtering

To ensure the reliability and validity of the collected questions, a multi\-stage quality assurance process was conducted\. The stages include quality assurance, filtration, hallucination\-driven QA selection, and domain taxonomy refinement\. Quality Assurance\.We manually reviewed all 1,500 QA pairs collected from the first phase to verify linguistic clarity, factual relevance, and domain appropriateness\. Five exclusion criteria have been defined to remove irrelevant questions\. First, questions that were ambiguous or could be interpreted in multiple ways were excluded to ensure clarity\. Second, opinion\-based questions that did not have a single verifiable factual answer were removed\. Third, questions with conflicting answers across reliable sources were filtered out to avoid uncertainty in the ground truth\. Fourth, questions with unreliable, unverifiable, or weakly supported ground\-truth references were excluded\. Fifth, knowledge\-based questions that lacked factual grounding or unsupported claims were removed to maintain the factual integrity of the dataset\. Filtration\.We filtered out duplicate questions, and any questions not fully written in MSA, including those partially expressed in regional dialects, were manually rewritten into MSA to ensure linguistic consistency\. We also rephrased some questions to enhance linguistic diversity and adversarial potential\. The questions were phrased in multiple styles, such as direct, riddle\-like formulations and indirect reasoning\. The quality assurance and filtration stages resulted in the exclusion of 347 questions\. Hallucination\-Driven QA Selection\.To ensure that the dataset focuses on hallucination\-prone questions, we evaluated the hallucination of 17 LLMs \([5\.1](https://arxiv.org/html/2605.17007#S5.SS1)\) on the 1,153 QA pairs resulting from the quality assurance and filtration stages\. Questions where two or less models produced hallucinated responses were excluded, as these questions were unlikely to reliably evaluate hallucination behavior\. This process resulted in a pool of 327 QA pairs\.

Notably, this large\-scale selection stage provided important insights that guided the subsequent data expansion phase\. These observations helped us better understand which question characteristics are less likely to trigger hallucinations and, more importantly, how questions can be formulated to increase hallucination risk\. By analyzing the excluded questions, we observed that many questions were straightforward factual questions or did not require complex reasoning\. In contrast, hallucinations were more likely to occur in questions involving named entities \(e\.g\., people, locations, or organizations with similar names\), numerical facts \(such as dates, rankings, or statistics\), and questions requiring precise domain knowledge\. We also observed that specific question formulations can significantly increase the likelihood of hallucinations, particularly when questions include misleading assumptions, comparative phrasing, or require exact recall rather than general knowledge\. We therefore found that both the knowledge requirements and the linguistic structure of a question play important roles in triggering hallucination\. Domain Taxonomy Refinement\.After excluding some questions from the previous stages, we merged some domain categories to reduce overlap and improve conceptual clarity\. We mergedArtandEntertainmentbecause many questions about music, cinema, literature, and popular culture naturally span both categories\. We also unifiedHealthandPsychologyunderHealth, as many hallucination\-prone questions in these areas involve medical misconceptions, mental health beliefs, and cognitive biases, which can fall into the broader category of Health\. In addition, we mergedFinancewithEconomicsdue to their strong conceptual overlap in financial literacy, markets, and economic misconceptions\.

We excluded theLawcategory from the final domain taxonomy\. This is because legal knowledge is highly jurisdiction\-dependent and varies significantly across countries, particularly within the Arab region, where legal systems differ based on national legislation, regulatory structures, and interpretations of Islamic law\. We found that including legal questions could introduce ambiguity when defining a single ground truth and reduce annotation consistency\. Therefore, we prioritized domains with more stable and universally verifiable factual knowledge\. Table[2](https://arxiv.org/html/2605.17007#S3.F2)lists the final set of hallucination categories defined inHalluScorewith illustrative examples\.

### 3\.3Dataset Expansion

To further strengthen the hallucination coverage in our dataset, we curated an additional 500 questions based on hallucination patterns observed in the hallucination\-driven evaluation stage, as explained in section[3\.2](https://arxiv.org/html/2605.17007#S3.SS2)\. Our analysis revealed that certain types of questions, such as false presuppositions, numerical traps, historical and cultural terms, and long\-tail knowledge, consistently triggered hallucination across different models\. To improve coverage of these failure modes, we designed new questions that target these weaknesses\. The additional questions were written without inspecting the specific responses generated by the evaluated models to avoid dataset contamination and bias toward particular model behaviors\.

During this process, we also focused on improving diversity in question formulation\. The newly curated questions include a mixture of direct factual questions, adversarially phrased questions, misleading formulations, and riddle\-style questions\. This linguistic variation was intentionally introduced to reduce pattern memorization and better evaluate factual robustness under different prompting styles\. Additionally, we aimed to improve coverage across domains and hallucination types by prioritizing underrepresented categories identified in the initial dataset\. Although the dataset is not perfectly balanced, we attempted to avoid severe under\-representation that could bias evaluation\. This expansion process resulted in a more comprehensive dataset that captures a wider range of hallucination triggers, which improves its suitability for evaluating LLMs’ hallucination detection and mitigation in Arabic\.

### 3\.4Annotation Process

Following the dataset collection and expansion stages, we performed a structured annotation process to assign multiple labels to each of the 827 QA pairs of theHalluScoredataset\. These labels capture key categories, including question type, adversarial intent, reasoning requirements, historical relevance, Arabic cultural context, and domain knowledge\. In addition to these annotations, we also included supporting metadata for each QA pair, such as a verified ground\-truth reference link and an answer explanation to facilitate evaluation and analysis\.

This extensive annotation process enables fine\-grained analysis of hallucination behavior and provides insights into the structural weaknesses of LLMs\. Therefore, the dataset allows systematic investigation of the question characteristics that most frequently trigger hallucination responses\. Therefore, the dataset helps move beyond traditional QA evaluation and supports deeper analysis of factual reliability and robustness in recent LLMs\. Table[5](https://arxiv.org/html/2605.17007#S3.F5)outlines some examples from theHalluScoredataset\. Labels Definition\.For question type annotation, we created a taxonomy consisting of 13 categories representing common hallucination\-triggering question patterns, such asConfusion, Misconceptions, False presuppositions, and Knowledge\-based questions\. The defined hallucination categories are:Confusion, Conspiracies, False presuppositions, Identity, Knowledge, Misconceptions, Misquotation, Myth, Paranormal, Proverbs, Stereotypes, Subjectivity,andSuperstition\. We combined theConspiracies, MythandParanormaltypes into a new type, which we namedPseudoscience\. We also added theCalculationtype for math\-based questions\. Table[3](https://arxiv.org/html/2605.17007#S3.F3)explains each question type\. Domain knowledge labels followed the taxonomy described in Section[3\.2](https://arxiv.org/html/2605.17007#S3.SS2), where each question was assigned a primary domain based on the main knowledge area required to answer it\. This taxonomy is further supported by prior hallucination benchmarks such as TruthfulQA and HalluQA, which demonstrate that these question types are particularly prone to inducing hallucinations in LLMs\[[20](https://arxiv.org/html/2605.17007#bib.bib20),[44](https://arxiv.org/html/2605.17007#bib.bib44)\]\.

Figure 3:Samples from the Question types inHalluScorewith their definitions and examples\.![Refer to caption](https://arxiv.org/html/2605.17007v1/figures/Table_3.png)Adversarial labeling was performed using a binary scheme \(yes/no\), indicating whether the question was intentionally phrased to mislead the model or to encourage incorrect assumptions\. Questions containing implicit traps, misleading premises, or confusing formulations were labeled as adversarial\. Historical labeling was also performed using a binary scheme\. The questions were labeled historical if answering them requires knowledge of people, events, or facts that predate 1999\. Arabic cultural labels were assigned to questions involving traditions, customs, heritage, arts, literature, food, language usage, or culturally significant figures and practices\. Similarly, a reasoning label was assigned to questions requiring multi\-step inference, implicit knowledge connections, or logical deduction beyond direct factual recall\. Questions that can be answered through simple fact retrieval were labeled as non\-reasoning\. Dataset Metadata\.In addition to the annotation labels, each QA pair includes an explanation field that is automatically generated by Gemini\-2\.5\-flash\. Figure[4](https://arxiv.org/html/2605.17007#S3.F4)shows the prompt used to generate the explanation\. We generated the explanation to provide a concise justification for the ground\-truth answer, supporting downstream evaluation scenarios such as LLM\-as\-a\-judge settings\. Then, we manually reviewed these explanations to ensure their factual correctness, linguistic clarity, and consistency, as well as to verify that all explanations were written in Arabic\. Furthermore, each QA pair is associated with a verified ground\-truth reference link, typically from Wikipedia and government websites\. This is to makeHalluScoresuitable for retrieval\-based evaluation and to facilitate future approaches to hallucination mitigation, such as RAG\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x2.png)Figure 4:Prompt provided to Gemini\-2\.5\-Flash to generate explanations of answers inHalluScoreFigure 5:Samples from theHalluScoredataset\. Advr: Adversarial, Rsn: Reasoning, Cul: Arabic cultural\-oriented, Dom: Domain knowledge, and Hist: History\. Within the type column, FPS: False presupposition, PSD: Pseudoscience, KNG: Knowledge, MSQ: Misquotation, and CON: Confusion\. Within the Domain knowledge column, Ent: Entertainment, Soci: Sociology, Scn: Science, Rlg: Religion, and Geo: Geography\.![Refer to caption](https://arxiv.org/html/2605.17007v1/figures/Table_4.png)

## 4HalluScore Analysis

Table II:Descriptive statistics ofHalluScoreacross different types\. We report the total number of questions in each type \(\#QA\), the average number of words in questions \(Q len\) and answers \(A len\), and the percentages of adversarial \(Adv\), reasoning \(Rsn\), historical \(Hist\), and Arabic cultural \(Cul\) questions in each type\.Type\#QAQ lenA lenAdv \(%\)Rsn \(%\)Hist \(%\)Cul \(%\)Calculation3720\.972\.008\.111000\.000\.00Confusion13315\.262\.1793\.9875\.9427\.0761\.65False presupposition698\.683\.6110023\.1927\.5417\.39Identity307\.373\.671000\.000\.000\.00Knowledge3719\.292\.791\.6217\.8441\.2438\.54Misconception487\.716\.8158\.330\.0012\.5016\.67Misquotation2613\.922\.2723\.080\.0010080\.77Proverbs209\.708\.7515\.000\.000\.00100Pseudoscience408\.954\.3592\.500\.000\.0032\.50Stereotype338\.125\.551000\.000\.0027\.27Subjective207\.957\.251000\.000\.000\.00Total82710\.623\.4143\.5826\.6329\.0937\.24Overall data distribution\.To better understand the dataset’s characteristics, we analyze in Table[II](https://arxiv.org/html/2605.17007#S4.T2)the distribution of theHalluScoredataset, including the number of questions, the average questions and answers lengths, and the proportion of each category binary label across all types, including adversarial intent, reasoning requirement, Arabic culture, and historical timeline\. As shown in Table[II](https://arxiv.org/html/2605.17007#S4.T2), the statistics reveal clear differences in linguistic complexity across types\.CalculationandConfusionquestions tend to be longer, with average question lengths of 20\.97 and 15\.26 words, respectively\. This reflects their higher reasoning demands\. In contrast,IdentityandMisconceptionquestions tend to be shorter but remain highly adversarial, indicating that hallucinations can be triggered even by short queries\. The statistics also show that a large proportion of the questions are adversarial and culturally relevant, reflecting our intention to stress\-test LLMs under challenging conditions that commonly trigger hallucinations, particularly in adversarial settings and culturally specific contexts\. Question type distribution\.Figure[6\(a\)](https://arxiv.org/html/2605.17007#S4.F6.sf1)illustrates the distribution of question types inHalluScore\. The dataset is dominated byKnowledgequestions \(44\.9%\), followed byConfusion\(16\.1%\) andFalse presuppositionquestions \(8\.3%\)\. These three categories constitute the majority of the dataset because our preliminary analysis, as discussed in Section[3\.2](https://arxiv.org/html/2605.17007#S3.SS2), showed that they are among the most likely to trigger hallucinations\. Knowledge questions primarily assess LLMs’ ability to answer long\-tail knowledge\.Confusionquestions challenge the model with misleading formulations to assess whether it can maintain factual accuracy under potentially confusing conditions, whereasFalse presuppositionquestions test whether the model can recognize that the question is based on false assumptions or not\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x3.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.17007v1/x4.png)\(b\)

Figure 6:Type and domain distribution across theHalluScoredataset\. \(a\) The type distribution across the questions\. \(b\) The knowledge domain proportion across the questions\.![Refer to caption](https://arxiv.org/html/2605.17007v1/x5.png)Figure 7:Normalized knowledge domain distribution across types\.Domain distribution\.Figure[6\(b\)](https://arxiv.org/html/2605.17007#S4.F6.sf2)illustrates the domain distribution in HalluScore\. The largest proportions of questions come from theEntertainment\(13\.9%\),Language\(12\.7%\),Health\(9\.3%\), andPolitics\(8\.8%\) domains, which reflect the dataset’s coverage of commonly encountered knowledge areas that require factual recall and domain\-specific terminologies\. A moderate distribution of 7% to 8% of the dataset comprisesReligion, Geography,andSociologyto further test the LLMs’ ability to answer historical and cultural questions\. Smaller but important domains, such asMathematics, are also included to introduce reasoning\-based question types\. Binary label statistics\.Table[II](https://arxiv.org/html/2605.17007#S4.T2)presents the distribution of categories binary labels across question types, whereas Figure[7](https://arxiv.org/html/2605.17007#S4.F7)illustrates how domain knowledge interacts with different question types\. From the type\-level analysis,False presupposition, Identity, Stereotype,andSubjectivequestions are entirely adversarial by design\. Furthermore,ConfusionandPseudosciencealso exhibit high levels of adversariality\. In contrast,Knowledgequestions exhibit a very low adversarial ratio \(1\.62%\) and it is used mainly as a baseline category to evaluate the model’s ability to answer factual questions correctly\.

Reasoning requirements are concentrated primarily inCalculation\(100%\) andConfusion\(75\.94%\) questions\. This reflects their design to evaluate whether models can maintain factual consistency when logical reasoning is required\. The domain analysis in Figure[7](https://arxiv.org/html/2605.17007#S4.F7)shows that theCalculationquestions are in theMathematicsdomain, presenting questions related to math, and religion, representing the Zakat \(form of obligatory charity in Islam\) calculation\.Confusion\-basedquestions span multiple domains, such asHealth, Language,andSociology, demonstrating their role as robustness tests across domains\.

Historical dependency is most prominent inMisquotationquestions \(100%\) andKnowledgequestions \(41\.24%\), reflecting their reliance on temporal facts, historical figures, or factual verification\. As shown in Figure[7](https://arxiv.org/html/2605.17007#S4.F7), these types frequently appear in domains such asPolitics, Religion,andEntertainment, which often require accurate historical grounding to avoid hallucinations\.

Cultural relevance is particularly high in numbers inProverbs\(100%\) andMisquotation\(80\.77%\) questions\. This highlights the importance of culturally grounded knowledge in hallucination evaluation, which is further supported by Figure[7](https://arxiv.org/html/2605.17007#S4.F7)\. These question types are strongly associated withLanguage, Religion,andSociologydomains\. Similarly,Confusion\-based questions show substantial cultural relevance \(61\.65%\), which indicates that cultural context can increase ambiguity and hallucination risk\. Type and domain association\.The domain\-level analysis in Figure[7](https://arxiv.org/html/2605.17007#S4.F7)reveals clear relationship between question types and knowledge areas\. For example,Stereotypequestions are strongly associated withSociology,Calculationquestions withMathematics, andIdentityquestions withTechnology\-related knowledge\. On the other hand,Subjectivequestions show more diverse domain coverage, reflecting their focus on opinion\-based rather than fact\-based responses\. Multi\-label interactions\.To better understand how multi\-labels co\-occur, we analyze in Figure[8](https://arxiv.org/html/2605.17007#S4.F8)the interactions between the four binary attributes: adversarial intent, reasoning requirement, historical dependency, and Arab cultural relevance\. As shown in Figure[8](https://arxiv.org/html/2605.17007#S4.F8), historical and cultural labels exhibit one of the strongest interactions, with 151 questions belonging to both categories\. This reflects the nature of many culturally grounded questions in Arabic contexts, which often involve historical figures, traditional sayings, or culturally significant events\. Adversarial questions constitute a large portion of the dataset, with 360 instances\. This reflects the dataset’s emphasis on evaluating hallucination robustness under challenging conditions\. A notable overlap exists between adversarial and reasoning questions \(121 instances\), which indicates that many adversarial questions require logical verification rather than simple factual recall\. Similarly, 128 adversarial questions also involve cultural knowledge, indicating that culturally grounded adversarial questions serve as an important hallucination trigger\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x6.png)Figure 8:Co\-occurrence matrix of binary labels inHalluScore\. Each cell\(i,j\)\(i,j\)represents the number of questions that simultaneously exhibit two categories \(adversarial intent, reasoning requirement, historical knowledge, and Arabic cultural\-oriented context\) at a time\. The diagonal values indicate the total number of questions associated with each individual label, while off\-diagonal values quantify pairwise co\-occurrence, revealing how often different categories appear together within the same question in theHalluScoredataset\.
## 5Experimental Setup

### 5\.1Evaluated models

A wide range of Arabic, multilingual, and reasoning\-based LLMs are evaluated on the HalluScore dataset\. A total of four Arabic LLMs were evaluated, including Allam\-preview\-7b\-instruct\[[7](https://arxiv.org/html/2605.17007#bib.bib7)\], Fanar\-1\-9b\[[9](https://arxiv.org/html/2605.17007#bib.bib9)\], Jais\-6\.7b\[[8](https://arxiv.org/html/2605.17007#bib.bib8)\], and Noon\-7b\[[56](https://arxiv.org/html/2605.17007#bib.bib56)\]\. Moreover, a total of eight multilingual models were evaluated, including Claude\-sonnet 4\.5\[[57](https://arxiv.org/html/2605.17007#bib.bib57)\], Deepseek\-v3\[[58](https://arxiv.org/html/2605.17007#bib.bib58)\], Grok\-4\[[59](https://arxiv.org/html/2605.17007#bib.bib59)\], GPT\-4\[[60](https://arxiv.org/html/2605.17007#bib.bib60)\], GPT\-5\[[61](https://arxiv.org/html/2605.17007#bib.bib61)\], Llama\-4\-Maverick\-17B\-128E\-Instruct\-FP8\[[62](https://arxiv.org/html/2605.17007#bib.bib62)\], Qwen3\-Next\-80B\-A3B\-Instruct\[[63](https://arxiv.org/html/2605.17007#bib.bib63)\], and Qwen3\-235B\-A22B\-Instruct\-2507\-tput\[[64](https://arxiv.org/html/2605.17007#bib.bib64)\]\. Furthermore, a total of five reasoning\-based models were evaluated, including Claude\-opus\-4\[[65](https://arxiv.org/html/2605.17007#bib.bib65)\], Deepseek\-r1\[[66](https://arxiv.org/html/2605.17007#bib.bib66)\], Grok\-4\-fast\-reasoning\[[59](https://arxiv.org/html/2605.17007#bib.bib59)\], GPT\-o3\[[67](https://arxiv.org/html/2605.17007#bib.bib67)\], and GPT\-o4\-mini\[[67](https://arxiv.org/html/2605.17007#bib.bib67)\]\.

### 5\.2Evaluation Setup

All models were evaluated in a zero\-shot question\-answering setting\. Each question was directly provided to the models without additional context or retrieval augmentation\. For API\-based models \(OpenAI, Claude, Together AI, Grok\), responses were obtained through their official APIs, while open\-source models were loaded locally using the HuggingFace Transformers library\. Model loading and inference were handled through a unified interface to ensure consistent evaluation conditions across all models\. To ensure fair comparison, deterministic decoding was used whenever possible by setting the temperature to zero and disabling sampling\. For Together AI and API\-based models, we also controlled generation parameters such as maximum token limits and repetition penalties to reduce verbosity and improve answer consistency\. To standardize responses, we used the following unified prompt to instruct the models to provide short and realistic answers in Arabic:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/prompts_halluscore.png)"Answer the following question in Arabic realistically, briefly, and without additional details: text: Answer:"\. This prompt was designed to reduce verbosity and discourage speculative explanations, allowing the evaluation to focus on factual correctness rather than generation style\. All implementation details are available in our GitHub repository\.

### 5\.3Evaluation protocol

In this study, we evaluated the factual consistency and faithfulness of LLMs’ outputs\. Factuality hallucination occurs when the generated response contains information that contradicts verifiable real\-world knowledge or established facts\[[2](https://arxiv.org/html/2605.17007#bib.bib2)\]\. On the other hand, faithfulness hallucination refers to the model’s failure to adhere to the context, which is the role and question provided within the user’s prompt in our case\.\[[2](https://arxiv.org/html/2605.17007#bib.bib2)\]\.

#### Human Evaluation\.

Human evaluation was used as the primary reference for assessing hallucination when evaluating model performance onHalluScore\. We manually annotate the models’ outputs against the corresponding ground\-truth\. Each answer was labeled for factuality and faithfulness hallucination\. Moreover, in cases where the main answer was correct, we also evaluated partial hallucinations, which occur when the LLM introduces additional incorrect or unsupported details in its response\.

To distinguish between hallucination and error, we follow the following criteria: hallucination is the generation of an unsupported real\-world fact, whereas an error is any other incorrect output\. Accordingly, all hallucinations are errors, but not all errors are hallucinations\. If a model responds with "I don’t know" or produces an incorrect answer while expressing uncertainty, hesitation, or lack of knowledge, the response is classified as an error rather than a hallucination, since the model does not present the information as factual\. In contrast, when an LLM generates an incorrect or fabricated answer and presents it with high confidence instead of expressing uncertainty, this behavior is considered a hallucination\. The following criteria are followed by the annotators to annotate LLM responses:

- •If the generated answer contradicts verified factual knowledge sources, then it is labelledfactual hallucination\.
- •If the answer incorrectly interprets the question, then it is labelledfaithfulness hallucination\.
- •If a question is ambiguous, unknown, or inherently unanswerable, but the model responds with a fabricated fact, then it is labelledfactual hallucination\.
- •If the answer includes statements, such as "many think", or "many people believe", then it is labelledfactual hallucination\.
- •If the answer includes statements, such as “this question cannot be answered”, or “I don’t know”, then it is labelledno hallucination\.
- •If the LLM answers cautiously, then it is regarded asno hallucination\.
- •If the answer is not fluent, but contains the correct answer, then it is labelledno hallucination\.
- •If a non\-hallucination answer consists of partially inaccurate information, label it asno hallucinationand write the full hallucination span\.

## 6Results and Discussion

### 6\.1Quantitative Evaluation

To evaluate hallucination rates, we relied on binary annotations \(0: non\-hallucinated, 1: hallucinated\) for factual and faithful hallucinations\. On the other hand, we specified hallucination spans for partial hallucination\. For each model, the factual and faithful hallucination rates were computed as the proportion of responses labeled as hallucinated\. Likewise, for partial hallucinations, we count the number of times a model generated them in its responses\. Table[III](https://arxiv.org/html/2605.17007#S6.T3)outlines the results of the evaluated LLMs on the HalluScore benchmark\. Overall, factual hallucinations constitute the dominant error type across all models, while faithfulness and partial hallucinations occur less frequently\.

Table III:Hallucination rates of the evaluated LLMs on theHalluScoredataset\. For each metric, thelowestandhighesthallucination rates are highlighted \(lower is better\)\.ModelAvg L\.Faith%Fact%Prt\.%Adv\.%Cult\.%Hist\.%Reas\.%Allam8\.993\.8768\.680\.0175\.0069\.1667\.9268\.64Claude Opus24\.550\.8533\.010\.0334\.7245\.4537\.5026\.82Claude Sonnet21\.651\.2135\.790\.0338\.8952\.2742\.5030\.91DeepSeek R121\.913\.8757\.680\.0360\.8369\.8161\.6750\.91DeepSeek V319\.592\.5454\.780\.0262\.5066\.2355\.4246\.82Fanar10\.8710\.0479\.320\.0180\.8384\.0984\.1780\.91GPT\-57\.900\.7325\.150\.0131\.1133\.4429\.1720\.91GPT\-4o10\.140\.8541\.230\.0144\.7247\.0842\.0843\.18GPT\-o35\.203\.5180\.050\.0081\.6786\.3683\.3377\.73GPT\-o4\-mini8\.691\.3346\.430\.0251\.1159\.7452\.9235\.91Grok413\.850\.9765\.420\.0465\.2879\.2270\.8362\.73Grok4 reasoning6\.780\.9752\.960\.0155\.5669\.1657\.5042\.73Jais8\.197\.1380\.410\.0190\.5675\.9777\.9280\.91Llama4 Maverick10\.141\.3357\.560\.0170\.5663\.3156\.2551\.36Noon12\.2213\.3085\.010\.0182\.2288\.6490\.4289\.09Qwen3 80B9\.521\.8161\.430\.0358\.6177\.6067\.9260\.91Qwen3 235B11\.350\.7359\.490\.0153\.6176\.9567\.0861\.36#### Models Hallucination\.

As shown in Table[III](https://arxiv.org/html/2605.17007#S6.T3), the best\-performing model, GPT\-5, achieved the lowest factual and faithfulness hallucination percentages of 25\.15% and 0\.73%, respectively, followed by Claude Opus with percentages of 33\.01% and 0\.85%, respectively\. The results demonstrate their strong capability to align with real\-world knowledge and user instructions\. Claude Sonnet also shows a strong factual robustness, with a factual hallucination percentage of 35\.79%\. However, it attained a higher faithfulness hallucination percentage when compared to GPT\-5 and Claude Opus\.

Noon, Jais, and GPT\-o3 exhibit the highest factual hallucination rates, exceeding 80%, which indicates that those models face difficulty in maintaining factual accuracy on Arabic hallucination\-prone questions\. Moreover, unlike the results obtained in AraHalluEval\[[3](https://arxiv.org/html/2605.17007#bib.bib3)\], where Allam showed comparative performance to multilingual and reasoning\-based models, such as GPT\-4o and Deepseek R1, it scored a higher factual hallucination percentage of 68\.68%\. This suggests that Allam appears more vulnerable when encountering questions designed to probe hallucination behavior\. This discrepancy further highlights the importance of stress\-testing models using hallucination\-focused benchmarks, as general evaluation datasets may overestimate factual reliability when they do not sufficiently challenge models with ambiguity, reasoning traps, or adversarial phrasing\.

#### Partial Hallucination\.

As shown in Table[III](https://arxiv.org/html/2605.17007#S6.T3), partial hallucination rates remain relatively low across most evaluated models, generally below 0\.05%\. This indicates that models tend to provide either fully correct or fully incorrect answers, rather than partially correct ones\. However, some models, such as Grok4, Claude Opus, and Claude Sonnet, exhibit slightly higher partial hallucination rates than others, suggesting a greater tendency to elaborate beyond the necessary answer\.

A notable pattern emerges when comparing partial hallucination with answer length\. Models that generate longer responses and attain low factual hallucination percentages, such as Claude Opus \(24\.55 words\) and Claude Sonnet \(21\.65 words\), tend to show higher partial hallucination percentages than models producing shorter answers, such as GPT\-o3 \(5\.20 words\) and GPT\-5 \(7\.90 words\)\. This trend suggests that increased wordiness may increase the likelihood of introducing unsupported details\. This highlights the importance of balancing completeness and factual precision in model responses, particularly in hallucination\-sensitive evaluation settings\. Figure[11](https://arxiv.org/html/2605.17007#S6.F11)illustrates partial hallucination samples from some LLMs\. We notice that numerical and named\-entity hallucinations are the most prevalent in partial hallucination\.

#### Hallucination per Type\.

Figure[9](https://arxiv.org/html/2605.17007#S6.F9)illustrates the distribution of factual hallucinations across question types for all evaluated models\. We applied normalization to prevent bias from uneven category sizes and show which question types most frequently trigger factual hallucinations across models\. Overall, there is a clear difference in how models behave depending on the nature of the question\.Confusion\- and False presupposition\-based questions induce hallucination across all models, which indicates that misleading phrasing and false premises remain an effective mechanism for triggering factual errors even among advanced models\. On the other hand,Knowledge\-based questions produce the lowest hallucination percentages, suggesting that models remain relatively reliable when answering straightforward factual queries that do not require complex reasoning or premise verification

Arabic LLMs, including Allam, Fanar, Jais, and Noon exhibit higher hallucination levels inCalculation, MisconceptionPseudosciencequestions\. This suggests that Arabic LLMs still face difficulties in numerical reasoning, correcting false beliefs, and rejecting scientifically unsupported claims\. Multilingual models, such as DeepSeek, Grok, Llama4\-Maverick, and Qwen, generally exhibit more balanced hallucination distributions across question types\. However, Deepseek models show relatively higher proportions of hallucinations inIdentity, Misquotation, Pseudoscience,andStereotypequestions, whereas Llama4\-Maverick exhibits increased hallucinations in theStereotypeandSubjectivecategories\. Similarly, the evaluated Grok models show a high percentage of hallucinations onMisquotation, Proverbs,andSubjectivityquestions\. This indicates that these multilingual models face challenges responding to questions related to cultural knowledge, quotation attribution, and socially framed claims\. The evaluated Qwen models show a similar behavior, with a higher percentage of hallucinations inKnowledge\-related questions\. This can be attributed to their small size when compared to other models\.

The GPT and Claude families generally exhibit lower proportions of hallucinations across most question types compared to several other models\. In particular, Claude Opus and Claude Sonnet show relatively stable behavior, with lower contributions to hallucination inCalculation, Misconception,andKnowledgequestions\. However, similar to other models, they remain susceptible toConfusionandFalse presuppositionquestions, indicating that adversarial phrasing continues to challenge even highly capable systems\. Within the GPT family, clear improvements can be observed across model generations\. Earlier models such as GPT\-4o and GPT\-o4\-mini exhibit moderate hallucination levels across several categories, particularly inSubjective, Confusion,andFalse presuppositionquestions\. In contrast, GPT\-5 consistently exhibits lower proportions of hallucinations across most question types, indicating improved factual grounding and response calibration\. This progression suggests that newer GPT models have become more robust to adversarial and reasoning\-heavy queries, although hallucinations still persist in questions containing misleading assumptions or ambiguous phrasing\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x7.png)Figure 9:Stacked distribution of factual hallucination percentages per model across question types\. The values are computed as:\(\# factual hallucinations for a model in a given question type\) / \(total number of questions in that type\) × 100\. The segments are stacked horizontally, such that the length of each segment reflects the hallucination rate for that category, and the full bar shows the distribution of hallucination behavior across all question types for that model\.
#### Hallucination in adversarial questions\.

As shown in table[III](https://arxiv.org/html/2605.17007#S6.T3), adversarial questions consistently produce higher hallucination rates compared to the overall factual hallucination scores\. Models such as Jais \(90\.56%\), Noon \(82\.22%\), GPT\-o3 \(81\.67%\), and Fanar \(80\.83%\) show particularly high vulnerability in adversarial settings\. This indicates that misleading or carefully crafted questions remain effective at exposing factual weaknesses\. In contrast, GPT\-5 \(31\.11%\) and Claude Opus \(34\.72%\) demonstrate the lowest hallucination rates under adversarial conditions, which suggests stronger robustness to adversarial questions\.

#### Hallucination in reasoning\.

Table[III](https://arxiv.org/html/2605.17007#S6.T3)reveals that for reasoning\-based questions, hallucination rates remain high across most models, particularly for Noon, Jais, Fanar, and GPT\-o3, all exceeding 75%\. This suggests that reasoning complexity remains a major source of hallucinations even when the required knowledge to calculate the result is available\. GPT\-5 \(20\.91%\) again shows the lowest hallucination rate in this category, followed by Claude Sonnet and Claude Opus, which indicates improved reasoning capability compared to most other models\.

#### Hallucination in Arabic culture\.

For cultural questions, as shown in Table[III](https://arxiv.org/html/2605.17007#S6.T3), most models show increased hallucination rates compared to their overall factual hallucination scores\. This trend is particularly evident in Grok4, Qwen, and DeepSeek models, which suggests that culturally grounded knowledge remains a challenge even for multilingual models\. Arabic LLMs, such as Fanar and Noon, also exhibit high hallucination rates in this category, indicating that language specialization alone does not guarantee robustness to culturally nuanced factual questions\.

#### Hallucination in history\.

As shown in Table[III](https://arxiv.org/html/2605.17007#S6.T3), historical questions follow a similar pattern to adversarial questions, with most models exhibiting elevated hallucination rates\. Noon \(90\.42%\), GPT\-o3 \(83\.33%\), and Fanar \(84\.17%\) show the highest vulnerability, indicating difficulty in maintaining temporal accuracy and historical grounding\. In contrast, GPT\-5 \(29\.17%\) and Claude Opus \(37\.50%\) again demonstrate comparatively stronger performance, suggesting better temporal knowledge consistency\.

#### Cross\-Model consistency\.

Figure[10](https://arxiv.org/html/2605.17007#S6.F10)presents the overlap percentage of factual hallucinations between pairs of models, indicating how frequently different models hallucinate on the same questions\. Higher values suggest stronger agreement in hallucination behavior\. Overall, the results indicate that hallucinations are not entirely random across models\. Instead, many models tend to hallucinate on a shared subset of challenging questions, while stronger models, such as GPT\-5, exhibit more distinct and limited hallucination patterns\.

Within the same model families, strong consistency is observed\. The Qwen models exhibit one of the highest overlaps, with Qwen\-80B and Qwen\-235B reaching 69\.8%, indicating that they tend to hallucinate on highly similar sets of questions\. The Claude models also demonstrate high consistency, with Claude Opus and Claude Sonnet sharing a 61\.6% overlap\. Similarly, DeepSeek models \(DeepSeek\-R1 and DeepSeek\-V3\) exhibit strong agreement with a 65\.8% overlap, suggesting that models within the same family tend to fail on similar questions\. A similar trend is evident in the Grok models, where Grok4 and Grok4\-Reasoning show a relatively high overlap \(66\.8%\), indicating that adding reasoning capabilities does not substantially alter hallucination patterns across the question in HalluScore\.

Across different model families, several notable patterns emerge\. Models with higher hallucination rates, such as GPT\-o3, Jais, Noon, and Fanar, tend to show larger overlaps with many other models, indicating that certain questions consistently trigger hallucinations across LLMs\. In contrast, GPT\-5 shows the lowest overlap with most LLMs, which is often below 35%, suggesting that it hallucinates on fewer, more distinct questions than other models\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x8.png)Figure 10:Heatmap of pairwise Jaccard similarity between models based on factual hallucination instances\. Each model is represented by the set of questions for which it produced a factual hallucination\. Similarity is computed asJ​\(A,B\)=\|A∩B\|\|A∪B\|J\(A,B\)=\\frac\{\|A\\cap B\|\}\{\|A\\cup B\|\}\.![Refer to caption](https://arxiv.org/html/2605.17007v1/x9.png)Figure 11:Examples of partial hallucination responses generated by some LLMs onHalluScore\. Partial hallucination spans are highlighted in yellow\.

### 6\.2Qualitative Analysis of Model Responses

Beyond quantitative evaluation, we perform a qualitative analysis of LLM responses to better understand the underlying causes of hallucinations\. By examining model outputs across different question types, we identify recurring weaknesses that contribute to hallucination, including failures in logical validation, cultural understanding, linguistic analysis, and knowledge consistency\. This analysis provides deeper insights into why LLMs are hallucinating\.

#### Anthropomorphism\.

From a design perspective, LLM anthropomorphism is the intentional use of human characteristics within a model’s interactions, such as pronouns, personality, backstory, or social roles, to create the illusion of a human partner as opposed to a computer program\[[68](https://arxiv.org/html/2605.17007#bib.bib68),[69](https://arxiv.org/html/2605.17007#bib.bib69)\]\. When a model is prompted to answer questions about inherently human traits, such as"What is your gender?", and responds with"male", it assigns itself a fictitious human characteristic\. This kind of response suggests a potential new term,anthropomorphism hallucination, to describe cases where the model fabricates human\-like intrinsic traits, such as gender, age, emotions, and beliefs\. This behavior, however, is not always undesirable and may sometimes be beneficial depending on the interaction context\. For example, when a user asks a question such as![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r1.png)"What is an opinion that you have personally changed your mind about?", and the model responds with![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r2a.png)![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r2b.png)"I used to think that success depended solely on hard work and diligence, but I later realized that circumstances, opportunities, and relationships also play a major role, and that luck and timing are important factors that cannot be ignored\.", the response may enhance the conversational experience by making the interaction feel more natural and engaging\. In such cases, anthropomorphic responses can improve user engagement, trust, and perceived empathy, particularly in conversational or educational settings where human\-like dialogue can make interactions more intuitive\[[68](https://arxiv.org/html/2605.17007#bib.bib68)\]\.

#### Reality violation\.

We identify a specific hallucination behavior that we refer to asreality violation\. It occurs when a model generates an answer to a question that is based on an impossible invalid scenario\. Instead of understanding the premise, the model focuses on answering a part of the question, which indicates a failure in premise validation\. For example:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r3.png)"Is it valid to fast on the 10th of Muharram if it coincides with Eid?"This question contains a logical inconsistency because the 10th of Muharram \(Ashura\) and the Islamic Eid days cannot occur on the same date in the Hijri calendar\. A reliable model should therefore identify the impossibility of the scenario before attempting to answer the jurisprudential aspect of the question\. However, some models ignore this contradiction and directly answer whether fasting is permissible, demonstrating a reality\-violation hallucination\. Similarly, in the following question:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r4.png)"If the sun rises at sunset, do doctors recommend wearing sunglasses?"Here, the premise itself violates basic physical reality\. Models that proceed to discuss medical recommendations about sunglasses without identifying the impossibility of the situation demonstrate a failure to verify real\-world feasibility\. This suggests that the model prioritizes generating a plausible response over validating whether the scenario itself is logically or physically possible\.

#### Reasoning length\.

Analyzing LLMs’ responses reveals that DeepSeek\-R1 and DeepSeek\-V3 tend to produce longer responses, as shown in Figure[12](https://arxiv.org/html/2605.17007#S6.F12), particularly on reasoning\-intensive questions\. This suggests that these models rely on more explicit intermediate reasoning steps to reach an answer\. In contrast, models such as Claude often reach correct answers with significantly shorter responses, suggesting more concise reasoning strategies or more efficient use of internal knowledge\. This difference suggests that while DeepSeek models emphasize detailed reasoning traces, other models may achieve similar outcomes through more compact answer generation\. These differences in response length and reasoning style may also explain hallucination patterns\. Models that generate longer reasoning traces may expose themselves to higher hallucination risk due to increased token generation and intermediate factual dependencies\. In contrast, models producing concise responses may rely more on calibrated internal knowledge, reducing opportunities for speculative generation\. This suggests that reasoning style may be an important factor influencing hallucination behavior beyond model accuracy alone\.

![Refer to caption](https://arxiv.org/html/2605.17007v1/x10.png)Figure 12:Example of response style differences between Claude Sonnet and DeepSeek\-R1 on a reasoning question\.
#### Explaining Proverbs

LLMs struggle with Arabic culturally grounded expressions, such as proverbs and idiomatic sayings\. In such cases, correct interpretation requires cultural and contextual knowledge rather than literal translation\. For example:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r5.png)"Someone told me: “Redden your eyes, puff your cheeks, and hold your hands\.” What does this mean?"The intended meaning of this proverb is:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r6.png)Suppressing anger and avoiding rushing into punishment\.However, the evaluated Arabic models failed to capture this cultural meaning\. For instance, Allam gives the following answer:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r7.png)"This is a popular expression meaning that he helped or served me\.", whereas Fanar generates the following response:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r8.png)It means that you are angry\.

These responses demonstrate different types of cultural hallucination\. Allam produces an unrelated interpretation that is not grounded in the proverb’s meaning, while Fanar captures only the emotional component \(anger\) but fails to identify the intended moral meaning related to restraint and self\-control\. This suggests that even Arabic\-specialized models may lack sufficient exposure to culturally specific idiomatic knowledge or may rely on literal semantic cues rather than contextual cultural understanding\.

#### Unfamiliarity with cultural terms\.

Our analysis of the LLM responses using theHalluScoredataset reveals that LLMs fail to answer questions related to historical cultural terminologies, particularly terms associated with traditional professions, regional dialects, or practices that are no longer commonly used\. These types of questions require specialized cultural knowledge that may not be sufficiently represented in modern training corpora\. For example:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r9.png)"What is the name of the medium\-sized Shaari fish in Bahrain?"![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r10.png)"What is the payment called that a pearl diver receives in the middle of the diving season?"These questions require knowledge of traditional Gulf terminology related to fishing and pearl diving, which are culturally significant but may not frequently appear in recent corpora\.

#### Question paraphrasing effect\.

We observe that small variations in question phrasing can significantly affect hallucination behavior\. Even when two questions express the same underlying concept, LLM responses may differ depending on how the question is formulated, suggesting sensitivity to surface\-level linguistic cues rather than purely factual understanding\. For example:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r11.png)"Will I get a disease if I crack my fingers?"![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r12a.png)![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r12b.png)"Is it possible to develop arthritis after cracking my fingers?"Although both questions relate to the common misconception that cracking fingers causes health problems, some models responded correctly to the first question by stating that no disease occurs, while incorrectly answering the second question by suggesting a possible link to arthritis\. This phenomenon suggests that hallucinations may not only stem from knowledge gaps but also from prompt sensitivity, where the inclusion of specific medical terminology \(e\.g\., arthritis\) increases the likelihood of the model generating a plausible but incorrect explanation\.

#### Grammar understanding\.

LLMs demonstrate weaknesses in answering questions related to Arabic grammatical analysis \(I’rab\) and classical Arabic literature\. These tasks require precise linguistic knowledge and familiarity with traditional Arabic grammar and poetry, which appear to be underrepresented in model training data\. For example,![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r13.png)"I liked the tree, its branches\.” What type of adjective \(na‘t\) appears in the sentence?"The correct answer is:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r14.png)"causal adjective \(na’t sababi\)\."However, models often fail to identify the correct grammatical category\. Similarly, models struggle with classical Arabic poetry analysis\. For instance,![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r15.png)"The verse: Indeed the eyes whose corners are wide… on which poetic meter \(bahr\) is this verse written?"\. The correct answer is:![[Uncaptioned image]](https://arxiv.org/html/2605.17007v1/figures/r16.png)"Al\-Basit meter\."These examples highlight that while models may perform well on modern language tasks, they remain less reliable when dealing with classical grammar, rhetorical analysis, and poetic structures that require specialized linguistic expertise\.

## 7Limitations

Despite the structured design and comprehensive evaluation of HalluScore, several limitations should be acknowledged\. First, although the dataset is carefully curated to include hallucination\-prone questions, its size remains moderate compared to large\-scale QA benchmarks, potentially limiting coverage of rare domains and edge cases\. Second, the dataset contains a strong focus on Arabic cultural, historical, and linguistic knowledge, which may limit direct comparability with general\-domain hallucination benchmarks and may bias evaluation toward culturally grounded knowledge\. Third, some question categories contain fewer samples, which may affect statistical comparisons across categories\.

From an evaluation perspective, despite the use of clear annotation guidelines, hallucination labeling may still involve subjective judgment, particularly for partial hallucinations and culturally nuanced responses\. Furthermore, the evaluation primarily focuses on factual and faithfulness hallucinations and does not explicitly measure reasoning trace correctness, calibration, or explanation quality\. The evaluation is also limited to a closed\-book generation setting and does not assess hallucination behavior in retrieval\-augmented or tool\-augmented generation scenarios\. Finally, the evaluation relies on a fixed prompting setup, and model performance may vary under different prompt formulations or decoding strategies\.

Despite these limitations, HalluScore provides a structured, multidimensional benchmark for evaluating hallucination risks in Arabic QA\. Future work may expand the dataset scale, incorporate retrieval\-based evaluation settings, explore prompt robustness, and analyze reasoning\-level hallucinations to further improve the comprehensiveness of hallucination evaluation\.

## 8Conclusion

This paper introducedHalluScore, the first Arabic QA benchmark specifically designed to evaluate hallucination in LLMs\. The dataset contains 827 QA pairs covering diverse domains and hallucination\-inducing scenarios\. It includes adversarial questions, cultural and historical knowledge, reasoning tasks, and linguistically complex queries\. Through extensive human annotation of 17 LLMs responses, including Arabic, multilingual, and reasoning\-oriented models, we provided a comprehensive analysis of hallucination behavior across different question types and knowledge domains\. Our results show that hallucinations are strongly influenced by question formulation, cultural grounding, and reasoning complexity\. Models such as GPT\-5 and Claude demonstrate comparatively lower hallucination rates, whereas all other evaluated LLMs remain vulnerable to adversarial phrasing, false presuppositions, and culturally specific knowledge\. Multilingual LLMs show difficulties in interpreting idiomatic expressions, historical terminology, and linguistic analysis tasks, such as grammar and poetic meter identification\. In addition to the weaknesses exhibited by multilingual LLMs, Arabic LLMs also show notable weaknesses in numerical reasoning and pseudoscientific claims\. Furthermore, our qualitative response analysis reveals several recurring failure patterns, including reality violation, cultural misunderstanding, paraphrase sensitivity, and grammatical analysis failures\.

These findings highlight that hallucination in Arabic LLMs is not solely a factual knowledge problem but also a challenge involving cultural competence, linguistic reasoning, and logical validation\. By providing both quantitative benchmarks and qualitative response analysis,HalluScoreoffers a structured framework for studying hallucination risks in Arabic QA systems\. Future work may extend this benchmark by expanding dataset size and introducing dialectal samples\.

## Acknowledgments

We would like to express our gratitude to Reem Aljunaid for her support in data collection and her contributions to the annotation process\. We would also like to thank Maryam Ahmed Alabdullatif, Maryam Abdullah Alabdullatif, Dr\. Noof Alabdullatif, Rawa Alturaif, Ahmed Alabdullatif, Osama Alabdullatif, and Ibrahim Alarfaj for their contributions to data collection\.

## References

- \[1\]J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald, “On faithfulness and factuality in abstractive summarization,” in*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2020, pp\. 1906–1919\.
- \[2\]L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin*et al\.*, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,”*ACM Transactions on Information Systems*, vol\. 43, no\. 2, pp\. 1–55, 2025\.
- \[3\]A\. Alansari and H\. Luqman, “Arahallueval: A fine\-grained hallucination evaluation framework for arabic llms,” in*Proceedings of The Third Arabic Natural Language Processing Conference*, 2025, pp\. 148–161\.
- \[4\]Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung, “Survey of hallucination in natural language generation,”*ACM computing surveys*, vol\. 55, no\. 12, pp\. 1–38, 2023\.
- \[5\]S\. Qi, L\. Gui, Y\. He, and Z\. Yuan, “A survey of automatic hallucination evaluation on natural language generation,”*arXiv preprint arXiv:2404\.12041*, 2024\.
- \[6\]A\. Alansari and H\. Luqman, “Large language models hallucination: A comprehensive survey,”*arXiv preprint arXiv:2510\.06265*, 2025\.
- \[7\]M\. S\. Bari, Y\. Alnumay, N\. A\. Alzahrani, N\. M\. Alotaibi, H\. A\. Alyahya, S\. AlRashed, F\. A\. Mirza, S\. Z\. Alsubaie, H\. A\. Alahmed, G\. Alabduljabbar*et al\.*, “Allam: Large language models for arabic and english,”*arXiv preprint arXiv:2407\.15390*, 2024\.
- \[8\]N\. Sengupta, S\. K\. Sahu, B\. Jia, S\. Katipomu, H\. Li, F\. Koto, W\. Marshall, G\. Gosal, C\. Liu, Z\. Chen*et al\.*, “Jais and jais\-chat: Arabic\-centric foundation and instruction\-tuned open generative large language models,”*arXiv preprint arXiv:2308\.16149*, 2023\.
- \[9\]F\. Team, U\. Abbas, M\. S\. Ahmad, F\. Alam, E\. Altinisik, E\. Asgari, Y\. Boshmaf, S\. Boughorbel, S\. Chawla, S\. Chowdhury*et al\.*, “Fanar: An arabic\-centric multimodal generative ai platform,”*arXiv preprint arXiv:2501\.13944*, 2025\.
- \[10\]M\. Mashaabi, S\. Al\-Khalifa, and H\. Al\-Khalifa, “A survey of large language models for arabic language and its dialects,”*arXiv preprint arXiv:2410\.20238*, 2024\.
- \[11\]A\. Alzubaidi, S\. Alsuwaidi, B\. E\. A\. Boussaha, L\. AlQadi, O\. Alkaabi, M\. Alyafeai, H\. Alobeidli, and H\. Hacid, “Evaluating arabic large language models: A survey of benchmarks, methods, and gaps,”*arXiv preprint arXiv:2510\.13430*, 2025\.
- \[12\]A\. Farghaly and K\. Shaalan, “Arabic natural language processing: Challenges and solutions,”*ACM Transactions on Asian Language Information Processing \(TALIP\)*, vol\. 8, no\. 4, pp\. 1–22, 2009\.
- \[13\]N\. Y\. Habash,*Introduction to Arabic natural language processing*\. Morgan & Claypool Publishers, 2010\.
- \[14\]H\. Mubarak, H\. Al\-Khalifa, and K\. S\. Alkhalefah, “Halwasa: Quantify and analyze hallucinations in large language models: Arabic as a case study,” in*Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\)*, 2024, pp\. 8008–8015\.
- \[15\]S\. Abdaljalil, H\. Kurban, and E\. Serpedin, “Halluverse25: Fine\-grained multilingual benchmark dataset for llm hallucinations,”*arXiv preprint arXiv:2503\.07833*, 2025\.
- \[16\]M\. Y\. Mohammed, S\. A\. Ali, S\. K\. Ali, A\. A\. Majeed, and E\. H\. Mohamed, “Aftina: enhancing stability and preventing hallucination in ai\-based islamic fatwa generation using llms and rag,”*Neural Computing and Applications*, pp\. 1–26, 2025\.
- \[17\]H\. Mubarak, R\. Malhas, W\. Mansour, A\. Mohamed, M\. Fawzi, M\. Hawasly, T\. Elsayed, K\. M\. Darwish, and W\. Magdy, “Islamiceval 2025: The first shared task of capturing llms hallucination in islamic content,” in*Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks*, 2025, pp\. 480–493\.
- \[18\]J\. Li, X\. Cheng, W\. X\. Zhao, J\.\-Y\. Nie, and J\.\-R\. Wen, “Halueval: A large\-scale hallucination evaluation benchmark for large language models,”*arXiv preprint arXiv:2305\.11747*, 2023\.
- \[19\]S\. Ramprasad, E\. Ferracane, and Z\. C\. Lipton, “Analyzing llm behavior in dialogue summarization: Unveiling circumstantial hallucination trends,” in*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2024, pp\. 12 549–12 561\.
- \[20\]Q\. Cheng, T\. Sun, W\. Zhang, S\. Wang, X\. Liu, M\. Zhang, J\. He, M\. Huang, Z\. Yin, K\. Chen*et al\.*, “Evaluating hallucinations in chinese large language models,”*arXiv preprint arXiv:2310\.03368*, 2023\.
- \[21\]X\. Liang, S\. Song, S\. Niu, Z\. Li, F\. Xiong, B\. Tang, Y\. Wang, D\. He, C\. Peng, Z\. Wang*et al\.*, “Uhgeval: Benchmarking the hallucination of chinese large language models via unconstrained generation,” in*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2024, pp\. 5266–5293\.
- \[22\]X\. Zhang, Z\. Liu, J\. Wang, H\. Zhang, F\. Xu, J\. Zhang, and X\. Wan, “C\-faith: A chinese fine\-grained benchmark for automated hallucination evaluation,” in*Proceedings of the 34th ACM International Conference on Information and Knowledge Management*, 2025, pp\. 6575–6579\.
- \[23\]H\. Ding, L\. Pang, Z\. Wei, H\. Shen, and X\. Cheng, “Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models,”*arXiv preprint arXiv:2402\.10612*, 2024\.
- \[24\]S\. AboulEla, P\. Zabihitari, N\. Ibrahim, M\. Afshar, and R\. Kashef, “Exploring rag solutions to reduce hallucinations in llms,” in*2025 IEEE International systems Conference \(SysCon\)*\. IEEE, 2025, pp\. 1–8\.
- \[25\]S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal, “Detecting hallucinations in large language models using semantic entropy,”*Nature*, vol\. 630, no\. 8017, pp\. 625–630, 2024\.
- \[26\]T\. Zhang, L\. Qiu, Q\. Guo, C\. Deng, Y\. Zhang, Z\. Zhang, C\. Zhou, X\. Wang, and L\. Fu, “Enhancing uncertainty\-based hallucination detection with stronger focus,” in*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023, pp\. 915–932\.
- \[27\]D\. Dale, E\. Voita, L\. Barrault, and M\. R\. Costa\-Jussà, “Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity even better,” in*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2023, pp\. 36–50\.
- \[28\]N\. Nonkes, S\. Agaronian, E\. Kanoulas, and R\. Petcu, “Leveraging graph structures to detect hallucinations in large language models,” in*Proceedings of TextGraphs\-17: Graph\-based Methods for Natural Language Processing*, 2024, pp\. 93–104\.
- \[29\]L\. Kong, Y\. Zhang, X\. Zhong, H\. Fu, Y\. Wang, and H\. Liu, “Halugnn: Hallucination detection in large language models using graph neural network,”*Expert Systems with Applications*, p\. 130857, 2025\.
- \[30\]S\. Dasgupta, S\. Nath, A\. Basu, P\. Shamsolmoali, and S\. Das, “Hallushift: Measuring distribution shifts towards hallucination detection in llms,” in*2025 International Joint Conference on Neural Networks \(IJCNN\)*\. IEEE, 2025, pp\. 1–8\.
- \[31\]P\. Manakul, A\. Liusie, and M\. Gales, “Selfcheckgpt: Zero\-resource black\-box hallucination detection for generative large language models,” in*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023, pp\. 9004–9017\.
- \[32\]J\. Zhang, Z\. Li, K\. Das, B\. Malin, and S\. Kumar, “Sac3: reliable hallucination detection in black\-box language models via semantic\-aware cross\-check consistency,” in*Findings of the Association for Computational Linguistics: EMNLP 2023*, 2023, pp\. 15 445–15 458\.
- \[33\]K\. Jiang, Q\. Zhang, D\. Guo, D\. Huang, S\. Zhang, Z\. Wei, F\. Ning, and R\. Li, “Ai\-generated news articles based on large language models,” in*Proceedings of the 2023 International Conference on Artificial Intelligence, Systems and Network Security*, 2023, pp\. 82–87\.
- \[34\]M\. Kim, H\. Jung, and M\.\-W\. Koo, “Self\-expertise: knowledge\-based instruction dataset augmentation for a legal expert language model,” in*Findings of the Association for Computational Linguistics: NAACL 2024*, 2024, pp\. 1098–1112\.
- \[35\]M\. Arslan, H\. Ghanem, S\. Munawar, and C\. Cruz, “A survey on rag with llms,”*Procedia computer science*, vol\. 246, pp\. 3781–3790, 2024\.
- \[36\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou*et al\.*, “Chain\-of\-thought prompting elicits reasoning in large language models,”*Advances in neural information processing systems*, vol\. 35, pp\. 24 824–24 837, 2022\.
- \[37\]S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. Weston, “Chain\-of\-verification reduces hallucination in large language models,” in*Findings of the association for computational linguistics: ACL 2024*, 2024, pp\. 3563–3578\.
- \[38\]M\. Hu, B\. He, Y\. Wang, L\. Li, C\. Ma, and I\. King, “Mitigating large language model hallucination with faithful finetuning,”*arXiv preprint arXiv:2406\.11267*, 2024\.
- \[39\]Y\.\-S\. Chuang, Y\. Xie, H\. Luo, Y\. Kim, J\. Glass, and P\. He, “Dola: Decoding by contrasting layers improves factuality in large language models,”*arXiv preprint arXiv:2309\.03883*, 2023\.
- \[40\]E\. Durmus, H\. He, and M\. Diab, “Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization,” in*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2020, pp\. 5055–5070\.
- \[41\]W\. Kryściński, B\. McCann, C\. Xiong, and R\. Socher, “Evaluating the factual consistency of abstractive text summarization,” in*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2020, pp\. 9332–9346\.
- \[42\]S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\.\-t\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi, “Factscore: Fine\-grained atomic evaluation of factual precision in long form text generation,” in*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023, pp\. 12 076–12 100\.
- \[43\]E\. Doostmohammadi, O\. Holmström, and M\. Kuhlmann, “How reliable are automatic evaluation methods for instruction\-tuned llms?” in*Findings of the Association for Computational Linguistics: EMNLP 2024*, 2024, pp\. 6321–6336\.
- \[44\]S\. Lin, J\. Hilton, and O\. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” in*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2022, pp\. 3214–3252\.
- \[45\]T\. Vu, M\. Iyyer, X\. Wang, N\. Constant, J\. Wei, J\. Wei, C\. Tar, Y\.\-H\. Sung, D\. Zhou, Q\. Le*et al\.*, “Freshllms: Refreshing large language models with search engine augmentation,” in*Findings of the Association for Computational Linguistics ACL 2024*, 2024, pp\. 13 697–13 720\.
- \[46\]B\. Goodrich, V\. Rao, P\. J\. Liu, and M\. Saleh, “Assessing the factual accuracy of generated text,” in*proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining*, 2019, pp\. 166–175\.
- \[47\]A\. El Ganadi, S\. Aftar, L\. Gagliardelli, F\. Ruozzi*et al\.*, “Generative ai for islamic texts: The eman framework for mitigating gpt hallucinations,” in*roceedings of the 17th International Conference on Agents and Artificial Intelligence\-ICAART*, vol\. 3, 2025, pp\. 1221–1228\.
- \[48\]M\. F\. Alghifari, M\. Kartiwi, M\. B\. A\. Zaim, and D\. O\. D\. Handayani, “Mitigating llm hallucinations in quranic content: An agentic approach using deployable language models,” in*2025 10th International Conference on Information and Communication Technology for the Muslim World \(ICT4M\)*\. IEEE, 2025, pp\. 1–6\.
- \[49\]R\. Vázquez, T\. Mickus, E\. Zosa, T\. Vahtola, J\. Tiedemann, A\. Sinha, V\. Segonne, F\. Sánchez\-Vega, A\. Raganato, J\. Libovickỳ*et al\.*, “Semeval\-2025 task 3: Mu\-shroom, the multilingual shared task on hallucinations and related observable overgeneration mistakes,”*arXiv preprint arXiv:2504\.11975*, 2025\.
- \[50\]D\. Dale, E\. Voita, J\. Lam, P\. Hansanti, C\. Ropers, E\. Kalbassi, C\. Gao, L\. Barrault, and M\. Costa\-jussà, “Halomi: A manually annotated benchmark for multilingual hallucination and omission detection in machine translation,” in*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023, pp\. 638–653\.
- \[51\]H\. Zhang, S\. Anjum, H\. Fan, W\. Zheng, Y\. Huang, and Y\. Feng, “Poly\-fever: A multilingual fact verification benchmark for hallucination detection in large language models,”*arXiv preprint arXiv:2503\.16541*, 2025\.
- \[52\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning, “Hotpotqa: A dataset for diverse, explainable multi\-hop question answering,” in*Proceedings of the 2018 conference on empirical methods in natural language processing*, 2018, pp\. 2369–2380\.
- \[53\]M\. Joshi, E\. Choi, D\. S\. Weld, and L\. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,” in*Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2017, pp\. 1601–1611\.
- \[54\]S\. Pandit, J\. Xu, J\. Hong, Z\. Wang, T\. Chen, K\. Xu, and Y\. Ding, “Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models,” in*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025, pp\. 2858–2873\.
- \[55\]A\. A\. Rahman, S\. Anwar, M\. Usman, I\. Ahmad, and A\. Mian, “Defan: Definitive answer dataset for llm hallucination evaluation,”*Information*, vol\. 16, no\. 11, p\. 937, 2025\.
- \[56\]Naseej for Technology, “Naseej launches its innovative arabic ai language model “noon” as an open\-source initiative,” Jun\. 19 2023, accessed: 2025\-07\-02\. \[Online\]\. Available:[https://naseej\.com/news/2023/06/](https://naseej.com/news/2023/06/)
- \[57\]Anthropic, “Introducing claude sonnet 4\.5,” 2025, accessed: 2026\-03\-18\. \[Online\]\. Available:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-5](https://www.anthropic.com/news/claude-sonnet-4-5)
- \[58\]A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan*et al\.*, “Deepseek\-v3 technical report,”*arXiv preprint arXiv:2412\.19437*, 2024\.
- \[59\]xAI, “Grok 4,” 2025, accessed: 2026\-03\-18\. \[Online\]\. Available:[https://x\.ai/news/grok\-4](https://x.ai/news/grok-4)
- \[60\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat*et al\.*, “Gpt\-4 technical report,”*arXiv preprint arXiv:2303\.08774*, 2023\.
- \[61\]A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram*et al\.*, “Openai gpt\-5 system card,”*arXiv preprint arXiv:2601\.03267*, 2025\.
- \[62\]Meta AI, “Llama\-4\-maverick\-17b\-128e\-instruct\-fp8,” https://ai\.azure\.com/catalog/models/Llama\-4\-Maverick\-17B\-128E\-Instruct\-FP8, 2025, azure AI Foundry model catalog\. Accessed: 2026\-03\-18\.
- \[63\]Alibaba Qwen Team, “Qwen3\-next\-80b\-a3b\-instruct,” 2025, qwen official blog\. Accessed: 2026\-03\-18\. \[Online\]\. Available:[https://qwen\.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd)
- \[64\]——, “Qwen3\-235b\-a22b\-instruct\-2507\-fp8,” 2025, together AI model catalog\. Accessed: 2026\-03\-18\. \[Online\]\. Available:[https://www\.together\.ai/models/qwen3\-235b\-a22b\-instruct\-2507\-fp8](https://www.together.ai/models/qwen3-235b-a22b-instruct-2507-fp8)
- \[65\]Anthropic, “System card: Claude opus 4 and claude sonnet 4,” Anthropic, Tech\. Rep\., 2025, accessed: 2026\-03\-18\. \[Online\]\. Available:[https://www\-cdn\.anthropic\.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47\.pdf](https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf)
- \[66\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi*et al\.*, “Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning,”*arXiv preprint arXiv:2501\.12948*, 2025\.
- \[67\]OpenAI, “Openai o3 and o4\-mini system card,” OpenAI, Tech\. Rep\., 2025, accessed: 2026\-03\-18\. \[Online\]\. Available:[https://cdn\.openai\.com/pdf/2221c875\-02dc\-4789\-800b\-e7758f3722c1/o3\-and\-o4\-mini\-system\-card\.pdf](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf)
- \[68\]M\. G\. Reinecke, F\. Ting, J\. Savulescu, and I\. Singh, “The double\-edged sword of anthropomorphism in llms,” in*Proceedings*, vol\. 114, no\. 1\. MDPI, 2025, p\. 4\.
- \[69\]C\. Sypherd, W\. Tang, and V\. Belle, “Breaking the illusion: Revisiting llm anthropomorphism,” in*The 4th International Conference on Human and Artificial Rationalities*\. Springer Nature, 2025, pp\. 1–19\.