How Much Do Legal RAG Systems Still Hallucinate?

arXiv cs.CL Papers

Summary

This research analyzes hallucination in legal RAG systems across eight models and two legal corpora, finding that hallucinations persist with rates ranging from under 10% to nearly half, particularly for false-premise questions.

arXiv:2608.14210v1 Announce Type: new Abstract: Hallucination is a major challenge for retrieval-augmented generation (RAG) systems in the legal domain, where ungrounded answers can lead to serious consequences. To better understand this problem, we conduct a fine-grained analysis of hallucination behavior in eight legal RAG systems across two legal corpora, the GDPR (in English) and a national civil law (in French). Using claim-level and answer-level evaluation, we report on hallucination density and severity, analyze performance across question categories and user personas, and validate our findings on an independent set of 142 legal-expert-authored questions. Our results show that hallucinations remain pervasive, ranging from less than 10% of responses for the best-performing systems to nearly half in the worst case. We further find that false-premise questions, containing incorrect assumptions that must be rejected, produce high hallucination rates on the manually-drafted questions.
Original Article
View Cached Full Text

Cached at: 08/17/26, 09:54 AM

# How Much Do Legal RAG Systems Still Hallucinate?
Source: [https://arxiv.org/html/2608.14210](https://arxiv.org/html/2608.14210)
###### Abstract

Hallucination is a major challenge for retrieval\-augmented generation \(RAG\) systems in the legal domain, where ungrounded answers can lead to serious consequences\. To better understand this problem, we conduct a fine\-grained analysis of hallucination behavior in eight legal RAG systems across two legal corpora, the GDPR \(in English\) and a national civil law \(in French\)\. Using claim\-level and answer\-level evaluation, we report on hallucination density and severity, analyze performance across question categories and user personas, and validate our findings on an independent set of 142 legal\-expert\-authored questions\. Our results show that hallucinations remain pervasive, ranging from less than 10% of responses for the best\-performing systems to nearly half in the worst case\. We further find that false\-premise questions, containing incorrect assumptions that must be rejected, produce high hallucination rates on the manually\-drafted questions\.

## 1Introduction

The growing reliance on generative artificial intelligence \(AI\) systems for knowledge acquisition has brought large language models \(LLMs\) to the forefront of modern information retrieval technologies\. Despite their impressive language understanding and generation capabilities, LLMs frequently producehallucinations, including factual inaccuracies, biased content, and flawed reasoning\. Such errors raise significant concerns about the reliability and trustworthiness of AI\-generated information[28](https://arxiv.org/html/2608.14210#bib.bib29)\. Consequently, hallucinations have become a major focus of research, attracting substantial attention from the AI community[15](https://arxiv.org/html/2608.14210#bib.bib22);[2](https://arxiv.org/html/2608.14210#bib.bib30)\.

Hallucinations pose significant risks in the legal domain, where inaccurate or ungrounded information can affect legal decisions leading to serious financial and societal consequences[4](https://arxiv.org/html/2608.14210#bib.bib4);[22](https://arxiv.org/html/2608.14210#bib.bib3);[1](https://arxiv.org/html/2608.14210#bib.bib31);[17](https://arxiv.org/html/2608.14210#bib.bib34)\. Several real\-world incidents have demonstrated these risks\. For example, a widely publicized legal case involved AI\-generated fictitious judicial decisions being cited in a legal filing, resulting in judicial sanctions[8](https://arxiv.org/html/2608.14210#bib.bib32)\. Moreover, a recent study reports that even legal research systems enhanced with retrieval\-augmented generation \(RAG\) remain prone to hallucinations, producing erroneous outputs in approximately one out of six queries[22](https://arxiv.org/html/2608.14210#bib.bib3)\. These findings highlight the pressing need for reliable methods to detect, evaluate, and mitigate hallucinations in legal AI systems\.

Addressing hallucinations in the legal domain is particularly challenging\. Legal reasoning often relies on large collections of heterogeneous legal sources, and determining the correct legal interpretation is often non\-trivial[24](https://arxiv.org/html/2608.14210#bib.bib33)\. As a result, hallucinations in legal AI systems cannot always be assessed using general\-purpose fact\-checking criteria, motivating the need for domain\-specific approaches to their analysis and mitigation[11](https://arxiv.org/html/2608.14210#bib.bib35);[22](https://arxiv.org/html/2608.14210#bib.bib3)\. Furthermore, hallucination behavior can vary substantially depending on the type of legal question\. Prior studies have shown that LLMs often fail to correct users’ incorrect legal assumptions in counterfactual question settings, suggesting that certain question categories are inherently more prone to hallucinations[4](https://arxiv.org/html/2608.14210#bib.bib4)\. Existing work also suggests that users with different backgrounds and goals interact with AI systems in different ways[19](https://arxiv.org/html/2608.14210#bib.bib36)\. Together, these observations motivate a systematic investigation of hallucination behavior across question categories and user personas\.

The widespread adoption of RAG further complicates hallucination analysis\. Although RAG is designed to reduce hallucinations by grounding responses in retrieved evidence, a hallucinated answer may originate from the retrieval component, the generation component, or their interaction[31](https://arxiv.org/html/2608.14210#bib.bib2);[5](https://arxiv.org/html/2608.14210#bib.bib24)\. Consequently, evaluating and diagnosing hallucinations in legal RAG systems is significantly more challenging than assessing standalone LLMs\.

Recent benchmarks have highlighted the prevalence of hallucinations in legal AI systems, including hallucinated citations[20](https://arxiv.org/html/2608.14210#bib.bib27)and responses that are unfaithful to retrieved evidence[36](https://arxiv.org/html/2608.14210#bib.bib25);[12](https://arxiv.org/html/2608.14210#bib.bib26)\. To obtain a more diagnostic understanding of these failures, evaluation must move beyond aggregate performance scores toward*claim\-level*analysis, where a claim is a minimal factual statement that can be independently verified against supporting evidence[25](https://arxiv.org/html/2608.14210#bib.bib9);[31](https://arxiv.org/html/2608.14210#bib.bib2)\. Recent work has adopted this perspective for RAG evaluation, factuality assessment, and hallucination detection by extracting claims from generated answers and assessing their support in the retrieved context[31](https://arxiv.org/html/2608.14210#bib.bib2);[13](https://arxiv.org/html/2608.14210#bib.bib1);[32](https://arxiv.org/html/2608.14210#bib.bib10)\. Such fine\-grained analysis enables a more detailed characterization of hallucination behavior\.

More recently, ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24)introduced a multilingual, multi\-jurisdiction benchmark for evaluating legal RAG systems through claim\-level assessment\. While the benchmark enables systematic comparison of state\-of\-the\-art systems, it provides limited insight into how hallucination behavior varies across question categories and user personas\. Moreover, it remains unclear whether the observed hallucination patterns generalize beyond the benchmark itself\. Consequently, our understanding of when and under which conditions legal RAG systems hallucinate remains incomplete\.

In this work, we address these gaps through a comprehensive analysis of hallucination behavior in legal RAG systems\. We examine hallucinations at both the*answer level*and the*claim level*, analyze their variation across question categories and user personas, and assess the generalizability of the observed patterns through external validation on manually\-drafted legal questions\. Answer\-level analysis measures how often hallucinations occur in generated responses, whereas claim\-level analysis captures the density and severity of hallucinated claims\. This distinction is particularly important in legal settings, where a response containing a single unsupported claim differs substantially from one containing multiple unsupported or contradicted claims\. By combining these complementary perspectives, we provide a more nuanced characterization of the prevalence, severity, and distribution of hallucinations in legal RAG systems\.

This paper makes the following contributions:

- •We present a comprehensive analysis of hallucination behavior in legal RAG systems\. In particular, we evaluate eight state\-of\-the\-art RAG systems on two legal sources provided by ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24): the General Data Protection Regulation \(GDPR\)[34](https://arxiv.org/html/2608.14210#bib.bib16)in English and a national civil law corpus \(CIVIL\) in French[9](https://arxiv.org/html/2608.14210#bib.bib17)\. The evaluated systems combine two retrievers, BM25[30](https://arxiv.org/html/2608.14210#bib.bib11)and E5[35](https://arxiv.org/html/2608.14210#bib.bib12), with four generators: GPT\-4[27](https://arxiv.org/html/2608.14210#bib.bib15), GPT\-5[33](https://arxiv.org/html/2608.14210#bib.bib19), Llama\-3\.1\-8B\-Instruct \(hereafter Llama3\-8B\)[10](https://arxiv.org/html/2608.14210#bib.bib14), and Mixtral\-8x7B[16](https://arxiv.org/html/2608.14210#bib.bib13)\. Our results show that hallucinations remain pervasive across all systems, ranging from 8% to 20% of responses for BM25\+GPT\-5 to nearly half of all responses for Llama3\-8B\-based systems\. We further show that most hallucinations are isolated single\-claim errors, while severe multi\-claim hallucinations occur less frequently but remain present even in top\-performing systems\.
- •We show that hallucination behavior in legal RAG systems is highly sensitive to both question category and user persona\. Across GDPR and CIVIL, BM25\+GPT\-5 consistently achieves the lowest hallucination rates, while Llama3\-8B\-based systems exhibit the highest rates\. We further find thatfalse\-premisequestions \(queries containing incorrect assumptions\) are particularly prone to hallucinations\. Moreover, hallucination rates vary across user personas, with questions associated with legal experts generally resulting in fewer hallucinations than those associated with civil officers or citizens, suggesting that query formulation, user intent, and domain knowledge play an important role in influencing the hallucination behavior of legal RAG systems\.
- •We validate our findings on an independent set of 142 manually\-drafted legal questions spanning GDPR and CIVIL\. The resulting trends are similar to the benchmark results, confirming that the observed hallucination patterns generalize beyond ClaimRAG\-LAW and are not benchmark\-specific\.

The remainder of the paper is organized as follows\. Section[2](https://arxiv.org/html/2608.14210#S2)reviews the related work on hallucination and legal RAG evaluation\. Section[3](https://arxiv.org/html/2608.14210#S3)describes the empirical setup and presents the hallucination analysis across systems\. Section[4](https://arxiv.org/html/2608.14210#S4)presents an external validation using manually drafted legal questions\. Section[5](https://arxiv.org/html/2608.14210#S5)discusses the threats to validity considerations and limitations of our study\. Finally, Section[6](https://arxiv.org/html/2608.14210#S6)concludes the paper\.

## 2Related Work

Hallucination detection has attracted considerable attention from the research community, with existing approaches broadly categorized asreference\-freeandreference\-basedmethods\. Reference\-free approaches assess outputs without relying on external knowledge sources, often leveraging self\-consistency measures[18](https://arxiv.org/html/2608.14210#bib.bib28)\. For example, SelfCheckGPT[23](https://arxiv.org/html/2608.14210#bib.bib18)detects hallucinations by measuring consistency across multiple outputs generated by the same model\. Similar ideas have recently been applied in the legal domain, where LLMs are used as factual\-consistency judges for legal question answering systems, demonstrating promising hallucination detection capabilities[14](https://arxiv.org/html/2608.14210#bib.bib20);[6](https://arxiv.org/html/2608.14210#bib.bib23)\.

In contrast, reference\-based approaches detect hallucinations by comparing model outputs against a trusted source of evidence, such as gold\-standard answers, retrieved documents, or supporting context\. As our work falls within this category, we focus on reference\-based techniques in the remainder of this section\.

A prominent class of reference\-based methods detects hallucinations through claim\-level verification\. Such methods first decompose generated responses into verifiable claims and then assess each claim against supporting evidence[25](https://arxiv.org/html/2608.14210#bib.bib9);[31](https://arxiv.org/html/2608.14210#bib.bib2);[7](https://arxiv.org/html/2608.14210#bib.bib21)\. One of the early examples is FActScore[25](https://arxiv.org/html/2608.14210#bib.bib9), which measures factual accuracy by verifying individual claims against retrieved evidence\. Building on this idea, FENICE[32](https://arxiv.org/html/2608.14210#bib.bib10)combines claim extraction and entailment analysis to assess factual consistency at multiple levels of granularity while identifying supporting evidence for each claim\. RefChecker[13](https://arxiv.org/html/2608.14210#bib.bib1)further structures claims as subject\-predicate\-object triples and evaluates their entailment against a reference context, making it particularly suitable for RAG systems\. Collectively, these methods establish claim\-level verification as an effective framework for fine\-grained hallucination analysis\.

Beyond general\-purpose hallucination detection, a growing body of research has focused on evaluating LLMs in the legal domain\. Early benchmarks, such as LexGLUE[3](https://arxiv.org/html/2608.14210#bib.bib7), LEXTREME[26](https://arxiv.org/html/2608.14210#bib.bib6), and LegalBench\-RAG[29](https://arxiv.org/html/2608.14210#bib.bib5), assess legal language understanding, reasoning, and retrieval capabilities across a range of tasks[21](https://arxiv.org/html/2608.14210#bib.bib8)\. More recently, attention has shifted toward the factual reliability of legal RAG systems\. Representative examples include LLeQA[21](https://arxiv.org/html/2608.14210#bib.bib8), which studies retrieve\-then\-read question answering over Belgian statutory law, and the work of[20](https://arxiv.org/html/2608.14210#bib.bib27), which evaluates legal hallucinations through citation verification in real\-world legal filings\. However, existing benchmarks rarely examine the generalizability of their findings beyond the benchmark\.

ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24)is a recent benchmark that provides a fine\-grained framework for evaluating hallucinations in legal RAG systems across multiple jurisdictions, languages, question categories, and user personas\. While it enables systematic comparison of legal RAG systems, its analysis primarily focuses on aggregate hallucination metrics, offering limited insight into how hallucination behavior varies across question categories and user perspectives\. Additionally, it does not examine whether the observed hallucination patterns persist beyond the benchmark questions\. Building on ClaimRAG\-LAW, we present a systematic analysis of hallucination behavior across eight state\-of\-the\-art RAG systems, considering different question categories, and personas, and further validate our findings on an independent set of legal\-expert\-authored questions\.

## 3Empirical Evaluation

This paper addresses the following research questions \(RQs\):

#### RQ1\. What is the hallucination rate of RAG systems in the legal domain?

This RQ provides an in\-depth analysis of the hallucination behavior for legal RAG systems on GDPR and CIVIL\. Specifically, we investigate hallucination both at the claim\-level as well as answer\-level to better understand whether hallucinations are distributed across responses or concentrated in a subset thereof\.

#### RQ2\. How do hallucination rates vary across question categories and user personas?

This RQ studies the impact of question type and user persona on the hallucination rates of legal RAG systems\.

### 3\.1Dataset

We conduct our analysis on ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24), a multilingual benchmark for fine\-grained evaluation of legal RAG systems\. It is composed of two sub\-datasets, collectively containing 317 expert\-validated question\-answer \(QA\) pairs derived from two legal sources: theGeneral Data Protection Regulation\(GDPR, English; 149 QA pairs\) and a national Civil Code \(CIVIL, French; 168 QA pairs\)\.

The benchmark covers four question categories:General Legal Research\(GR, 268 QA pairs, 84\.5%\),Factual Recall\(FR, 33 QA pairs, 10\.4%\),False Premise\(FP, 13 QA pairs, 4\.1%\), andJurisdiction/Time\-Specific\(JT, 3 QA pairs, 0\.9%\)\. It also considers three user personas with different expertise levels:Legal Expert\(LE, 156 QA pairs, 49\.2%\),Civil Officer\(CO, 106 QA pairs, 33\.4%\), andCitizen\(CI, 55 QA pairs, 17\.4%\)\.

### 3\.2Evaluated RAG Systems

We evaluate the same eight RAG systems as ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24), obtained by combining two retrievers \(BM25 and E5\) with four generators \(Llama3\-8B, Mixtral\-8x7B, GPT\-4, and GPT\-5\)\.

### 3\.3Evaluation Procedure

To answer our RQs, we analyze the generated answers of the RAG systems on GDPR and CIVIL datasets, following the same evaluation setup as ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24)\. For each question, we examine the generated answer, corresponding ground\-truth answer, extracted claims, entailment labels, question category, and persona without re\-executing the underlying RAG systems\.

We reportclaim\-level hallucination rate \(CL\), defined as the proportion of hallucinated claims among all generated claims, andanswer\-level hallucination rate \(AL\), defined as the proportion of answers containing at least one hallucinated claim \(i\.e\., a neutral or contradictory claim\)[31](https://arxiv.org/html/2608.14210#bib.bib2)\. Thus, CL measures hallucination density, whereas AL measures hallucination frequency\.

We additionally reportstrict claim\-level\(CLs\) andstrict answer\-level\(ALs\) hallucination rates, which count only explicitly contradicted claims as hallucinations[13](https://arxiv.org/html/2608.14210#bib.bib1)\. These stricter metrics provide a conservative estimate of hallucination behavior by focusing on factually incorrect legal claims rather than merely unsupported content\.

Finally, beyond the above metrics, we analyze the distribution of hallucinated claims within hallucinated answers to determine whether hallucinations are concentrated in a small number of severely affected answers or spread more evenly across a system’s responses\.

The evaluation materials and source code will be released upon publication under an appropriate open\-source license\.

### 3\.4Legal RAG Hallucination Rates \(RQ1\)

Table 1:Hallucination rates \(%\) of RAG systems in legal application context \(RQ1\)\.MMdenotes the average number of claims per response \(mean±\\pmstandard deviation\)\.↓\\downarrowindicates that lower values are better\.Table 2:Distribution of hallucinated claims per answer \(RQ1\)\.Table[1](https://arxiv.org/html/2608.14210#S3.T1)reports the hallucination rates for the evaluated RAG systems according to the metrics defined above\. We report the standard deviation \(SD\) values for CL and CLsbut not for AL and ALs, as the answer\-level metrics are essentially binary indicators of whether an answer is marked as hallucinated\. We also report the average number of claims per response \(columnMM\) to provide context for the claim\-level results\.

Hallucination Density\.The table shows that GPT\-based RAG systems consistently exhibit the lowest CL values, with BM25\+GPT\-5 achieving the best results on both GDPR \(1\.5%±3\.81\.5\\%\\pm 3\.8\) and CIVIL \(1\.2%±4\.71\.2\\%\\pm 4\.7\)\. In contrast, Llama3\-8B\-based systems generally produce the highest claim\-level hallucination rates, with E5\+Llama3\-8B on GDPR \(9\.5%±12\.49\.5\\%\\pm 12\.4\) and with BM25\+Llama3\-8B on CIVIL \(17\.9%±25\.817\.9\\%\\pm 25\.8\)\. These results show that hallucination can be highly impacted by the retrieval and generator components in a RAG system\. Furthermore, the relatively large SD values suggest that hallucinations may be concentrated in a subset of the answers rather than occurring uniformly across outputs, as discussed later in this section\.

The average number of claims per generated answer \(MM\) also varies across RAG systems\.MMprovides an indication of response verbosity\. For example, BM25\+GPT\-5 produces the highest average number of claims overall \(GDPR:17\.9±12\.217\.9\\pm 12\.2, CIVIL:7\.5±5\.67\.5\\pm 5\.6\)\. In contrast, E5\+Llama3\-8B generates substantially fewer claims compared to GPT\-based systems\. Despite their greater verbosity, GPT\-based systems maintain lower hallucination rates, suggesting that the higher hallucination rates of open\-weight models cannot be attributed solely to response length\.

Differences in verbosity also appear to depend on the retrieval component\. Across both datasets, E5\-based RAG systems consistently generate fewer claims per answer than BM25\-based ones, highlighting that verbosity could partially depend on the underlying retrieval\. The contexts retrieved by E5 appear to encourage generators to produce more conservative and concise answers, unlike BM25 retrieval, which generally encourages more elaborate responses\.

Furthermore, the gap between CL and CLsvalues reveals distinct hallucination patterns\. A large gap, as observed for E5\+GPT\-5 on GDPR \(5\.6%5\.6\\%vs\.1\.5%1\.5\\%\), suggests that most hallucinated claims are rather unsupported rather than explicitly contradicted by the evidence\. In contrast, a smaller gap, as observed for BM25\+Llama3\-8B on GDPR \(4\.6%4\.6\\%vs\.3\.5%3\.5\\%\), indicates a larger proportion of explicit contradicted claims\. Since contradicted claims correspond to factually incorrect legal information, they represent a more severe hallucination than simply neutral claims and would therefore require careful verification, ideally by legal experts\.

Hallucination Frequency\.The answer\-level results reveal that hallucinations remain common across all evaluated systems\. BM25\+GPT\-5 is the most robust against hallucinations, achieving the lowest AL hallucination rate on GDPR \(20\.120\.1%\) and CIVIL \(8\.338\.33%\)\. In contrast, E5\+Llama3\-8B performs worst on GDPR \(48\.348\.3%\), while BM25\+Llama3\-8B records the highest hallucination rate on CIVIL \(45\.845\.8%\)\. These results indicate that, in the worst case, BM25\+GPT\-5 hallucinates in roughly one out of five answers, whereas Llama3\-8B\-based systems hallucinate in nearly half of the generated answers\.

Taken together, CL and AL values provide a complementary view of hallucination behavior\. High values for both metrics indicate frequent and dense hallucinations \(many unsupported claims in many answers\), as observed for E5\+Llama3\-8B on GDPR \(AL =48\.348\.3%, CL =9\.59\.5%±12\.4\\pm 12\.4\) and BM25\+Llama3\-8B on CIVIL \(AL =45\.845\.8%, CL =17\.917\.9%±25\.8\\pm 25\.8\)\. In contrast, low AL with relatively high CL values indicate that hallucinations are concentrated in a smaller number of responses, e\.g\., E5\+GPT\-5 on CIVIL \(AL =22\.022\.0%, CL =4\.94\.9%, with a standard deviation of±12\.7\\pm 12\.7, indicating severe outliers\)\. conversely, high AL combined with low CL values suggest hallucinations are spread across many answers, but remain limited in each one \(i\.e\., only few unsupported claims\), e\.g\., BM25\+GPT\-4 on GDPR \(AL =30\.830\.8%, CL =2\.92\.9%\)\. This latter pattern is more common across the evaluated systems, suggesting that legal hallucinations often arise from isolated unsupported claims rather than entirely hallucinated answers\. The most desirable behavior is characterized by low values for both metrics, reflecting infrequent and limited hallucinations\.

#### Hallucination Severity\.

To better characterize hallucination severity, Table[2](https://arxiv.org/html/2608.14210#S3.T2)reports the distribution of unsupported claims per answer \(H\)\. Specifically, column H0 denotes hallucination\-free answers, and columns Hkk\(k≥1k\\geq 1\) denotes answers containing exactlykkunsupported claims\.

Across both datasets, H1 is the dominant category, indicating that most hallucinated answers contain only a single unsupported claim and are therefore localized rather than pervasive errors\. Consistent with the results above, BM25\+GPT\-5 ranks among the most reliable systems, achieving the highest number of hallucination\-free responses on CIVIL \(H0 = 154\) and tying for the highest count on GDPR \(H0 = 119\), alongside BM25\+Mixtral\-8x7B and E5\+GPT\-4\. In contrast, E5\+Llama3\-8B records the lowest H0 count on GDPR \(77\), while BM25\+Llama3\-8B performs worst on CIVIL \(91\)\.

Severe hallucinations involving multiple unsupported claims remain present even by the strongest systems, albeit less frequently\. For example, E5\+GPT\-5 occasionally produces answers containing eight or more unsupported claims \(H8 on GDPR and H9 on CIVIL in one answer each\)\. Although rare, such cases highlight the potential risk of high\-severity hallucinations in legal applications and motivate fine\-grained evaluation beyond aggregate metrics\.

Overall, the results suggest that hallucinations in legal RAG systems are more often isolated than pervasive, but severe multi\-claim hallucinations remain a persistent concern\.

### 3\.5Hallucination Rate Across Question Categories and Personas \(RQ2\)

Table 3:Hallucination rates \(%\) acrossquestion categories\(RQ2\)\. GR = General Research, FR = Factual Recall, and FP = False Premise\. CL and AL denote claim\-level and answer\-level hallucination rates, where±\\pmis the standard deviation\.Hallucinations Analysis by Question Category\.Table[3](https://arxiv.org/html/2608.14210#S3.T3)reports CL and AL across question categories \(GR: General Research, FR: Factual Recall, and FP: False Premise\)\. FR and FP results should be view as preliminary due to the limited number of questions, whereas the Jurisdiction/Time category is excluded from the analysis because of its very small sample size\. These categories capture distinct legal reasoning demands: FR questions primarily require retrieving explicit legal facts, GR questions require explaining or synthesizing legal rules and conditions, and FP questions require identifying and rejecting an incorrect assumption before answering\.

For the GR category, Llama3\-8B\-based systems consistently exhibit the highest hallucination rates\. E5\+Llama3\-8B performs worst on GDPR \(AL = 46\.6%\), whereas BM25\+Llama3\-8B has the highest hallucination rate on CIVIL \(AL = 39\.4%\)\. The relatively large standard deviations of the CL scores across both datasets indicate substantial variability in claim\-level hallucination behavior, suggesting that Llama3\-8B is particularly prone to hallucinations on GR questions regardless of the retriever used\.

At the other end of the spectrum, E5\+Mixtral\-8x7B and E5\+GPT\-4 achieve the lowest AL on GDPR \(20\.6%\), followed closely by BM25\+GPT\-5 \(22\.1%\)\. However, BM25\+GPT\-5 attains the lowest CL overall \(1\.6±3\.61\.6\{\\scriptstyle\\pm 3\.6\}\), with considerably lower variability than E5\+Mixtral\-8x7B \(9\.49\.4\) and E5\+GPT\-4 \(7\.17\.1\)\. Taken together, these results suggest that BM25\+GPT\-5 is the most robust system for GR questions, combining low hallucination frequency with stable claim\-level behavior, a trend that is further confirmed on CIVIL, where it achieves the lowest hallucination rates\.

For the FR category, Llama3\-8B\- and Mixtral\-8x7B\-based systems generally exhibit the highest hallucination rates across both GDPR and CIVIL\. Notable exceptions are BM25\+Mixtral\-8x7B on GDPR and E5\+Mixtral\-8x7B on CIVIL, both achieving an AL of 0%\. These results suggest that retrieval effectiveness may depend on corpus characteristics, including differences in language and document structure\. In contrast, GPT\-based generators consistently achieve low hallucination rates across both datasets and retrievers, indicating greater robustness to variations in retrieval quality\. Overall, while open\-weight models appear more sensitive to the choice of retriever and dataset, GPT\-based systems maintain comparatively stable performance\.

We observe for FP category a similar pattern as for FR\. Llama3\-8B and Mixtral\-8x7B generally exhibit the highest hallucination rates across both datasets, with BM25\+Mixtral\-8x7B having AL of100100% on CIVIL, i\.e\., every response to FP questions contained at least one hallucinated claim\. GPT\-5 remains comparatively robust, consistently achieving low hallucination rates and reaching00% in several settings\.

The FP category appears particularly challenging for legal RAG systems as it requires not only retrieving relevant legal information but also verifying an incorrect assumption\. For example, the question“How does GDPR require that controllers and processors submit their assessments and safeguards to a public registry for transparency?”contains a false premise, as the GDPR imposes no such public registry requirement\. Many systems implicitly accept the false premise and generate answers that build on it\. While GPT\-5 demonstrates resilience to this failure mode, the different results across systems call for further research to better understand the impact of question types on hallucination\.

Table 4:Hallucination rates acrosspersonas\(%\) \(RQ2\)\. LE = Legal Expert, CO = Civil Officer, and CI = Citizen\.Hallucination Analysis by User Persona\.Table[4](https://arxiv.org/html/2608.14210#S3.T4)reports CL and AL across three personas: Legal Expert \(LE\), Civil Officer \(CO\), and Citizen \(CI\)\. These personas capture distinct legal information needs\. CI questions are primarily fact\-oriented and focus on rights, obligations, and statutory references, e\.g\.,*“What is the minimum age at which a child’s consent is considered valid for the processing of personal data under the GDPR?”*\. CO questions emphasize compliance and institutional responsibilities, e\.g\.,*“What are the primary objectives of the GDPR regarding the protection of natural persons and the free movement of personal data?”*\. LE questions require legal interpretation and the application of legal provisions to more complex scenarios, e\.g\.,*“What specific conditions must be met for the transfer of personal data to a third country without an adequacy decision or appropriate safeguards?”*\.

Across both datasets, GPT\-based systems generally achieve the lowest hallucination rates, with BM25\+GPT\-5 emerging as the most consistent configuration\. In contrast, Llama3\-8B\-based systems typically exhibit the highest hallucination rates, particularly when combined with E5 retrieval\.

On GDPR, LE questions typically result in fewer hallucinations than CO questions, while some systems achieve their lowest rates on CI questions\. A similar pattern emerges on CIVIL, where differences between personas are substantially larger\. In particular, CO and CI questions prove challenging for several open\-weight models\. For example, BM25\+Llama3\-8B and BM25\+Mixtral\-8x7B exceed 70% AL for both personas, whereas GPT\-based systems remain below 20% in most cases and often below 5% with GPT\-5\.

Interestingly, LE questions frequently result in lower hallucination rates than CO questions despite requiring more sophisticated legal reasoning\. One possible explanation is that expert\-authored questions tend to be more precise and legally constrained, reducing ambiguity and limiting opportunities for unsupported inferences\. In contrast, CO questions often require reasoning about legal procedures and institutional responsibilities, encouraging broader explanations that may not be fully grounded in the retrieved evidence\.

Overall, the results suggest that legal RAG systems can be sensitive to changes in question formulation and user intent\.

Table 5:Hallucination rates \(%\) of RAG systems evaluated on manually curated legal expert questions within a legal application context\.MMrepresents the average number of claims per response, reported as mean±\\pmstandard deviation\.Table 6:Answer\-level hallucination rates \(%\) by question category on expert\-authored legal questions\.

## 4Validation on Expert\-Authored Questions

To assess the robustness of our findings beyond ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24), we conducted an external validation using 142 questions manually drafted by a third\-party legal expert, evenly split between GDPR and CIVIL\. Table[5](https://arxiv.org/html/2608.14210#S3.T5)reports the overall hallucination rates, while Table[6](https://arxiv.org/html/2608.14210#S3.T6)provides a breakdown by question category\. Overall, the principal trends observed on ClaimRAG\-LAW are reproduced\. GPT\-based systems continue to exhibit the lowest hallucination rates, whereas Llama3\-8B\-based systems are the most prone to hallucination\. Although the hallucination rates differ from those observed on the benchmark, the consistency of the system rankings across both legal corpora suggests that the observed patterns are not benchmark\-specific and generalize to independently curated legal questions\.

The category\-level results confirm the difficulty of FP questions\. Across both GDPR and CIVIL, FP questions frequently yield some of the highest answer\-level hallucination rates, indicating that many systems continue to accept incorrect assumptions rather than explicitly rejecting them\. This finding reinforces the importance of premise verification as a key challenge for legal RAG systems\.

The external validation also provides additional insight into the behavior of GPT\-4 and GPT\-5\. While the two models achieve broadly comparable overall performance, they exhibit distinct hallucination profiles\. GPT\-4 often attains slightly lower overall CL and AL hallucination rates, whereas GPT\-5 consistently achieves lower strict hallucination rates \(CLsand ALs\)\. This suggests that GPT\-5 is less prone to generating claims that directly contradict the available evidence, unlike GPT\-4\. Consequently, GPT\-5 appears more effective at avoiding severe hallucinations, even when the overall hallucination rates of the two models are similar\.

Taken together, these results indicate that improving retrieval quality alone is unlikely to mitigate hallucinations in legal RAG systems\. Future research should therefore focus not only on retrieval effectiveness, but also on improving faithfulness to retrieved evidence and strengthening premise\-verification capabilities, particularly for questions containing misleading or incorrect assumptions\.

## 5Threats to Validity and Limitations

Internal Validity\.Our analysis relies on the benchmark data and diagnostic metrics reported in ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24)\. Consequently, errors in the benchmark annotations or reported metrics may affect the hallucination rates and analyses presented in this study\. To mitigate this threat, we relied on the benchmark’s expert\-validated QA pairs and standardized evaluation pipeline, which ensure a consistent comparison across all analyzed RAG systems\.

External Validity\.Our study is limited to two legal sources \(GDPR and CIVIL\) and eight RAG systems\. Consequently, the findings may not generalize to other jurisdictions, languages, retrievers, or foundation models\. We partially mitigated this threat by evaluating systems across datasets from different jurisdictions and languages, as well as multiple retriever\-generator combinations\.

Limitations\.Our analysis is based on the benchmark outputs and therefore does not assess the impact of alternative prompts, chunking strategies, retrieval parameters, or newer model versions\. Furthermore, some question categories, particularly False Premise and Jurisdiction/Time\-Specific questions, contain relatively few examples, which may affect the stability of the corresponding results\. Future work should extend the evaluation to additional legal corpora, larger datasets, and expert\-reviewed hallucination annotations\.

## 6Conclusion

We have presented a fine\-grained analysis of hallucinations in eight legal retrieval\-augmented generation \(RAG\) systems evaluated on two legal corpora, GDPR and CIVIL\. Building on ClaimRAG\-LAW[5](https://arxiv.org/html/2608.14210#bib.bib24), our study has characterized hallucination behavior through complementary answer\-level and claim\-level analyses\.

Our results show that hallucinations remain a persistent challenge in legal RAG systems\. Hallucination rates vary substantially across configurations, ranging from the most robust system, BM25\+GPT\-5, to the least robust, Llama3\-8B\-based systems, which can hallucinate in nearly half of all generated responses\. We further find that most hallucinations consist of isolated unsupported claims rather than pervasive errors throughout an answer\. Moreover, hallucination behavior varies substantially across question categories and user personas, with factual\-recall and false\-premise questions, as well as queries posed by non\-expert users, proving particularly challenging\. By validating our findings on an independent set of expert\-authored legal questions, we provide evidence that the observed trends are not specific to ClaimRAG\-LAW and generalize beyond the benchmark\.

In future work, we plan to extend this analysis with legal\-specific evaluation metrics and additional diagnostic studies to better understand the factors contributing to hallucinations in legal RAG systems and to support the development of more reliable AI\-assisted legal applications\.

## References

- Banerjeeet al\.\(2025\)S\. Banerjee, A\. Agarwal, and S\. SinglaLLMs will always hallucinate, and we need to live with this\.InIntelligent Systems and Applications,pp\. 624–648\.Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p2.1)\.
- Banget al\.\(2025\)Y\. Bang, Z\. Ji, A\. Schelten, A\. Hartshorn, T\. Fowler, C\. Zhang, N\. Cancedda, and P\. FungHallulens: llm hallucination benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 24128–24156\.Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p1.1)\.
- Chalkidiset al\.\(2022\)I\. Chalkidis, A\. Jana, D\. Hartung, M\. Bommarito, I\. Androutsopoulos, D\. Katz, and N\. AletrasLexGLUE: a benchmark dataset for legal language understanding in english\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4310–4330\.External Links:[Link](https://doi.org/10.18653/v1/2022.acl-long.297)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p4.1)\.
- Dahlet al\.\(2024\)M\. Dahl, V\. Magesh, M\. Suzgun, and D\. E\. HoLarge legal fictions: profiling legal hallucinations in large language models\.Journal of Legal Analysis16\(1\),pp\. 64–93\.External Links:[Link](https://doi.org/10.1093/jla/laae003)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p2.1),[§1](https://arxiv.org/html/2608.14210#S1.p3.1)\.
- Daset al\.\(2026\)S\. Das, S\. Abualhaija, and D\. BianculliFine\-grained claim\-level rag benchmark for law\.External Links:2605\.21071,[Link](https://arxiv.org/abs/2605.21071)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.14210#S1.p4.1),[§1](https://arxiv.org/html/2608.14210#S1.p6.1),[§2](https://arxiv.org/html/2608.14210#S2.p5.1),[§3\.1](https://arxiv.org/html/2608.14210#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.14210#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2608.14210#S3.SS3.p1.1),[§4](https://arxiv.org/html/2608.14210#S4.p1.1),[§5](https://arxiv.org/html/2608.14210#S5.p1.1),[§6](https://arxiv.org/html/2608.14210#S6.p1.1)\.
- Enguehardet al\.\(2025\)J\. Enguehard, M\. Van Ermengem, K\. Atkinson, S\. Cha, A\. G\. Chowdhury, P\. K\. Ramaswamy, J\. Roghair, H\. R\. Marlowe, C\. S\. Negreanu, K\. Boxall,et al\.LeMAJ \(legal llm\-as\-a\-judge\): bridging legal reasoning and llm evaluation\.InProceedings of the Natural Legal Language Processing Workshop 2025,pp\. 318–337\.External Links:[Link](https://doi.org/10.18653/v1/2025.nllp-1.23)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p1.1)\.
- Galimzianovaet al\.\(2025\)D\. Galimzianova, A\. Boriskin, and G\. ArshinovFrom rag to reality: coarse\-grained hallucination detection via nli fine\-tuning\.InProceedings of the Fifth Workshop on Scholarly Document Processing \(SDP 2025\),pp\. 353–359\.External Links:[Link](https://doi.org/10.18653/v1/2025.sdp-1.34)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p3.1)\.
- GC AI \(2026\)GC AIAI hallucinations in legal cases: a sanctions tracker \(2026\)\.Note:https://gc\.ai/blog/ai\-hallucination\-legal\-casesAccessed: 2026\-07\-22Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p2.1)\.
- Grand\-Duché de Luxembourg \(2025\)Grand\-Duché de LuxembourgNational civil code \(consolidated version\)\.Journal officiel du Grand\-Duché de Luxembourg \(Legilux\)\.External Links:[Link](https://legilux.public.lu/)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Houet al\.\(2024\)A\. Hou, W\. Jurayj, N\. Holzenberger, A\. Blair\-Stanek, and B\. Van DurmeGaps or hallucinations? scrutinizing machine\-generated legal analysis for fine\-grained text evaluations\.InProceedings of the Natural Legal Language Processing Workshop 2024,Miami, FL, USA,pp\. 280–302\.External Links:[Link](https://aclanthology.org/2024.nllp-1.24/),[Document](https://dx.doi.org/10.18653/v1/2024.nllp-1.24)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p3.1)\.
- Huet al\.\(2026\)J\. Hu, Y\. Li, J\. Zhong, L\. Zou, and W\. QiDetecting hallucinations in retrieval\-augmented generation via semantic\-level internal reasoning graph\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 27826–27841\.Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p5.1)\.
- Huet al\.\(2024\)X\. Hu, D\. Ru, L\. Qiu, Q\. Guo, T\. Zhang, Y\. Xu, Y\. Luo, P\. Liu, Y\. Zhang, and Z\. ZhangRefChecker: reference\-based fine\-grained hallucination checker and benchmark for large language models\.External Links:2405\.14486,[Link](https://arxiv.org/abs/2405.14486)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p5.1),[§2](https://arxiv.org/html/2608.14210#S2.p3.1),[§3\.3](https://arxiv.org/html/2608.14210#S3.SS3.p3.1)\.
- Huet al\.\(2025\)Y\. Hu, L\. Gan, W\. Xiao, K\. Kuang, and F\. WuFine\-tuning large language models for improving factuality in legal question answering\.InProceedings of the 31st international conference on computational linguistics,pp\. 4410–4427\.Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p1.1)\.
- Huanget al\.\(2025\)L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.External Links:[Link](https://doi.org/10.1145/3703155)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p1.1)\.
- Jianget al\.\(2024\)A\. Q\. Jianget al\.Mixtral of experts\.External Links:2401\.04088,[Link](https://arxiv.org/abs/2401.04088)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Latif \(2025\)Y\. A\. LatifHallucinations in large language models and their influence on legal reasoning: examining the risks of ai\-generated factual inaccuracies in judicial processes\.Journal of Computational Intelligence, Machine Reasoning, and Decision\-Making10\(2\),pp\. 10–20\.External Links:[Link](https://morphpublishing.com/index.php/JCIMRD/article/view/2025-02-07)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p2.1)\.
- Liet al\.\(2024\)Q\. Li, J\. Geng, C\. Lyu, D\. Zhu, M\. Panov, and F\. KarrayReference\-free hallucination detection for large vision\-language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Findings,pp\. 4542–4551\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.262),[Link](https://aclanthology.org/2024.findings-emnlp.262)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p1.1)\.
- Linet al\.\(2025\)Z\. Lin, Z\. Wang, Y\. Pan, V\. Manjunatha, R\. A\. Rossi, A\. Lau, L\. Huang, and T\. SunPersona\-SQ: a personalized suggested question generation framework for real\-world documents\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\),pp\. 210–247\.External Links:[Link](https://aclanthology.org/2025.naacl-demo.20/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-demo.20)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p3.1)\.
- Liuet al\.\(2026\)P\. Liu, D\. Stammbach, and P\. HendersonWho checks the citations? benchmarking legal hallucination detection\.External Links:2606\.21155,[Link](https://arxiv.org/abs/2606.21155)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p5.1),[§2](https://arxiv.org/html/2608.14210#S2.p4.1)\.
- Louiset al\.\(2024\)A\. Louis, G\. van Dijck, and G\. SpanakisInterpretable long\-form legal question answering with retrieval\-augmented large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 22266–22275\.External Links:[Link](https://doi.org/10.1609/aaai.v38i20.30232)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p4.1)\.
- Mageshet al\.\(2025\)V\. Magesh, F\. Surani, M\. Dahl, M\. Suzgun, C\. D\. Manning, and D\. E\. HoHallucination\-free? assessing the reliability of leading AI legal research tools\.Journal of Empirical Legal Studies22\(2\),pp\. 216–242\.External Links:[Link](https://doi.org/10.1111/jels.12413)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p2.1),[§1](https://arxiv.org/html/2608.14210#S1.p3.1)\.
- Manakulet al\.\(2023\)P\. Manakul, A\. Liusie, and M\. GalesSelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 9004–9017\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.557)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p1.1)\.
- Mik \(2024\)E\. MikThe ground truth about legal hallucinations\.InICML 2024 Workshop on Generative AI and Law \(GenLaw\),Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p3.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12076–12100\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p5.1),[§2](https://arxiv.org/html/2608.14210#S2.p3.1)\.
- Niklauset al\.\(2023\)J\. Niklaus, V\. Matoshi, P\. Rani, A\. Galassi, M\. Stürmer, and I\. ChalkidisLEXTREME: a multi\-lingual and multi\-task benchmark for the legal domain\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 3016–3054\.External Links:[Link](https://doi.org/10.18653/v1/2023.findings-emnlp.200)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p4.1)\.
- OpenAIet al\.\(2024\)OpenAIet al\.GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Orgadet al\.\(2025\)H\. Orgad, M\. Toker, Z\. Gekhman, R\. Reichart, I\. Szpektor, H\. Kotek, and Y\. BelinkovLLMs know more than they show: on the intrinsic representation of LLM hallucinations\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 66880–66913\.Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p1.1)\.
- Pipitone and Alami \(2024\)N\. Pipitone and G\. H\. AlamiLegalBench\-rag: a benchmark for retrieval\-augmented generation in the legal domain\.External Links:2408\.10343,[Link](https://arxiv.org/abs/2408.10343)Cited by:[§2](https://arxiv.org/html/2608.14210#S2.p4.1)\.
- Robertson and Zaragoza \(2009\)S\. Robertson and H\. ZaragozaThe probabilistic relevance framework: BM25 and beyond\.Found\. Trends Inf\. Retr\.3\(4\),pp\. 333–389\.External Links:ISSN 1554\-0669,[Link](https://doi.org/10.1561/1500000019)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Ruet al\.\(2024\)D\. Ru, L\. Qiu, X\. Hu, T\. Zhang, P\. Shi, S\. Chang, C\. Jiayang, C\. Wang, S\. Sun, H\. Li,et al\.RAGChecker: a fine\-grained framework for diagnosing retrieval\-augmented generation\.Advances in Neural Information Processing Systems37,pp\. 21999–22027\.External Links:[Link](https://doi.org/10.52202/079017-0692)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p4.1),[§1](https://arxiv.org/html/2608.14210#S1.p5.1),[§2](https://arxiv.org/html/2608.14210#S2.p3.1),[§3\.3](https://arxiv.org/html/2608.14210#S3.SS3.p2.1)\.
- Scirèet al\.\(2024\)A\. Scirè, K\. Ghonim, and R\. NavigliFENICE: factuality evaluation of summarization based on natural language inference and claim extraction\.InFindings of the Association for Computational Linguistics ACL 2024,pp\. 14148–14161\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-acl.841)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p5.1),[§2](https://arxiv.org/html/2608.14210#S2.p3.1)\.
- Singhet al\.\(2026\)A\. Singhet al\.OpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- The European Parliament and the Council of the European Union \(2016\)The European Parliament and the Council of the European UnionRegulation \(eu\) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec \(General Data Protection Regulation\)\.External Links:[Link](https://eur-lex.europa.eu/eli/reg/2016/679/oj)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. WeiImproving text embeddings with large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11897–11916\.External Links:[Link](https://aclanthology.org/2024.acl-long.642/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.642)Cited by:[1st item](https://arxiv.org/html/2608.14210#S1.I1.i1.p1.1)\.
- Yehet al\.\(2026\)S\. Yeh, S\. Li, and T\. MallickLUMINA: detecting hallucinations in RAG system with context–knowledge signals\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=oJgNNBNEJM)Cited by:[§1](https://arxiv.org/html/2608.14210#S1.p5.1)\.

Similar Articles

RAG hallucinations are annoying AF

Reddit r/AI_Agents

The team discovered that 80% of RAG hallucinations were caused by poor retrieval, not the generation model, emphasizing the need to evaluate retrieval and generation separately to effectively debug bad answers.

RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration

arXiv cs.CL

RAGognizer introduces a hallucination-aware fine-tuning approach that integrates a lightweight detection head into LLMs for joint optimization of language modeling and hallucination detection in RAG systems. The paper presents RAGognize, a dataset of naturally occurring closed-domain hallucinations with token-level annotations, and demonstrates state-of-the-art hallucination detection while reducing hallucination rates without degrading language quality.