Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretation

# Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretation

Source: https://arxiv.org/html/2511.02626

Renfei Dang1∗, Peng Hu1∗, Zhejian Lai1, Changjiang Gao1, Min Zhang2, Shujian Huang1†

1National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
2Huawei Translation Services Center, Beijing, China

{dangrf,hup,laizj,gaocj}@smail.nju.edu.cn, [email protected], [email protected]

## Abstract

Prior works have shown that fine-tuning on new knowledge can induce factual hallucinations in large language models (LLMs), leading to incorrect outputs when evaluated on previously known information. However, the specific manifestations of such hallucination and its underlying mechanisms remain insufficiently understood. Our work addresses this gap by designing a controlled dataset—Biography-Reasoning—and conducting a fine-grained analysis across multiple knowledge types and two task types, including knowledge question answering (QA) and knowledge reasoning tasks. We find that hallucinations not only severely affect tasks involving newly introduced knowledge, but also propagate to other evaluation tasks. Moreover, when fine-tuning on a dataset in which a specific knowledge type consists entirely of new knowledge, LLMs exhibit elevated hallucination tendencies. This suggests that the degree of unfamiliarity within a particular knowledge type, rather than the overall proportion of new knowledge, is a stronger driver of hallucinations. Through interpretability analysis, we show that learning new knowledge weakens the model's attention to key entities in the input question, leading to an over-reliance on surrounding context and a higher risk of hallucination. Conversely, reintroducing a small amount of known knowledge during the later stages of training restores attention to key entities and substantially mitigates hallucination behavior. Finally, we demonstrate that disrupted attention patterns can propagate across lexically similar contexts, facilitating the spread of hallucinations beyond the original task.

## 1 Introduction

Large language models (LLMs) acquire rich factual knowledge during pre-training on massive text corpora, and are subsequently post-trained to follow human instructions and perform a wide range of downstream tasks.

However, during the Supervised Fine-Tuning (SFT) phase, models may encounter new knowledge not covered in pre-training. Prior research suggests that introducing new knowledge in the post-training phase increases the risk of factual hallucinations, where models generate fabricated yet plausible statements. This occurs because, when models learn new knowledge, they may erroneously generate related information in irrelevant contexts. These studies primarily focus on the effects within knowledge-intensive QA tasks during SFT, and we advance this line of research by investigating the fine-grained manifestations and underlying causes of hallucinations.

To support this investigation, we construct a controlled experimental dataset—Biography-Reasoning. The dataset is composed of biographical entities and their four attributes, which serve as four knowledge types. We further design twelve reasoning tasks using these knowledge. By controlling the proportion of known and unknown knowledge within different types and tasks in the training data, we systematically analyze the impact of learning new knowledge on hallucination risks.

Our experiments reveal that training on unknown knowledge significantly elevates hallucination risks in the same task, while also inducing non-negligible hallucination effects on other out-of-domain test tasks. Importantly, we further find that when a knowledge type consists entirely of new knowledge, even a small amount of such data can markedly increase hallucination tendencies. Through further interpretability analyses, we find that learning new knowledge significantly weakens the model's attention to key entities in the question, thereby triggering factual hallucinations. In contrast, training on known knowledge strengthens the model's attention to key entities. Motivated by this observation, we introduce a simple training method KnownPatch, which restores disrupted attention patterns by injecting a small amount of known knowledge during the later stages of training, and thus alleviates factual hallucinations. Finally, by constructing carefully designed variants of reasoning tasks, we demonstrate that lexical similarity (measured by token overlap between contexts), rather than semantic similarity of the contexts, is the primary driver of hallucination propagation across tasks.

The main contributions of this paper are:

- **Fine-Grained Analysis**: A detailed analysis across knowledge types and task types reveals the manifestations of new-knowledge-induced hallucinations, showing that when all knowledge within a specific type is entirely unknown, it is more likely to trigger severe hallucinations, even on unrelated QA test sets.

- **Mechanism Interpretability**: An analysis of attention mechanisms shows that learning new knowledge reduces attention to key question entities, causing hallucinations. In addition, lexically similar contexts facilitate the spread of these attention patterns, enabling cross-task hallucination effects.

## 2 Related Work

### New Knowledge and Hallucinations

Existing studies have indicated that introducing new knowledge into LLMs may trigger hallucinations. Subsequent works have provided deeper analyses of this phenomenon.

Gekhman et al. (2024) found that as the proportion of new knowledge in fine-tuning data increases, the model's hallucination tendency intensifies. Kang et al. (2024) analyzed that when fine-tuned LLMs encounter unknown queries during testing, their responses imitate those associated with unknown examples in the fine-tuning data. From the perspective of token probabilities, Sun et al. (2025) show that after learning new knowledge, the generation probabilities of answer entity tokens increase significantly even in irrelevant contexts, suggesting that the model may over-generalize newly acquired knowledge and consequently produce hallucinations.

However, previous studies focus mainly on closed-book QA settings with mixed knowledge types during training, while our controlled setup disentangles them to provide a more detailed analysis of new knowledge-induced hallucinations across types and tasks. Furthermore, we also investigate the underlying mechanisms of these phenomenon through an analysis of attention weights.

### Reducing Hallucinations

Numerous studies are currently exploring ways to mitigate model hallucinations. A common approach involves providing additional relevant context to the model to reduce hallucinations during generation, such as through retrieval from knowledge bases or leveraging other large models to generate context. Additionally, some research explicitly avoids hallucination risks by refusing to answer uncertain or unfamiliar questions.

In another direction, many studies encourage the model to generate more known knowledge from pre-training, for example, by promoting factual outputs via reinforcement learning or by training only on known knowledge during supervised fine-tuning to enhance the model. Our work builds on the SFT with known knowledge approach, but rather than pursuing comprehensive filtering across all training data, KnownPatch only introduces a small number of known knowledge samples in the later stages of training, and alleviates the model's tendency for hallucination.

## 3 Methodology of Analyzing Hallucinations

We aim to systematically investigate factual hallucinations in LLMs caused by learning different knowledge-related tasks. However, in real-world datasets, most factual knowledge may have already been seen by LLMs during pre-training, making it difficult to precisely control whether the knowledge being learned is new to the model. To address this limitation, we construct a synthetic dataset named Biography-Reasoning, which allows a controllable examination of hallucination behaviors under varying knowledge types and task types.

### 3.1 Biography-Reasoning Dataset

Following data construction methodologies, we design the Biography-Reasoning dataset. The dataset centers on individuals as the key entities, with each person associated with four attributes: birth year, death year, major, and university. We refer to the same attribute of different individuals as a knowledge type. Our dataset includes two types of knowledge-related tasks: knowledge QA and knowledge-based reasoning tasks.

For knowledge QA tasks, questions are formulated by directly querying one of the attributes given the person's name. Each task consists of questions on a single type, resulting in four QA tasks (e.g., Major_QA). For knowledge-based reasoning tasks, we design three types of chain-of-thought-requiring reasoning tasks. Specifically, these include:

- **Single Reasoning**: extracting one attribute from a single entity and performing a simple reasoning process;
- **Comparative Reasoning**: extracting one attribute from each of two entities and performing comparative reasoning between them;
- **Novel Reasoning**: extracting one attribute from a single entity and performing a newly defined reasoning task, such as mathematical or symbolic reasoning.

Table 1 presents examples of the constructed questions. The reasoning tasks are intentionally designed to be more complex than mere knowledge extraction as QA problems. Some of them require auxiliary knowledge (e.g., the major Dentistry belongs to the field Medicine), which the model is expected to contain. To further guarantee the model's proficiency, we additionally collect and train on these auxiliary facts.

For each knowledge type we construct one QA and three reasoning tasks, leading to a total of 4 QA and 12 reasoning tasks per individual.

### 3.2 Controlled Study Design

To examine factual hallucinations caused by training with tasks containing new knowledge, we need to discriminate **known** and **unknown** knowledge, control their usage during training, and evaluate related hallucinations. Since initially the model has no exposure to any knowledge of our synthetic dataset, we prepare the study by continue pre-training the model with a subset of the knowledge, which becomes **known** to the model; and keep another subset of the knowledge as **unknown**. By mixing the constructed questions from known and unknown knowledge in varying proportions, we are able to create situations where different proportion of newly introduced knowledge participates in training.

To evaluate how training leads to hallucinations, we reserve another subset of knowledge as **test** knowledge. The test knowledge are continue pre-trained together with the known knowledge during the preparation, but are kept away from further training. Therefore, the difference in performance on test set with and without unknown knowledge in the training data indicates the influence of factual hallucinations induced by training new knowledge.

In addition, we use the real-world ENTITYQUESTIONS dataset derived from Wikidata (denoted as Wiki) as an out-of-distribution (OOD) test set to provide a more robust evaluation.

### 3.3 Models and Setups

We conduct experiments primarily using the Qwen2.5-1.5B model. As supplementary validation, we also perform key experiments on Llama3.2-1B, Qwen3-8B-Base, and Qwen2.5-32B to assess generalization across model scales and architectures, with their results provided in Appendix G.

As our experiments are conducted on base models, we first apply SFT to endow them with the ability to answer questions in the evaluation sets. For QA analysis, SFT is conducted solely on knowledge QA data, whereas for reasoning,