Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator

# Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator
Source: [https://arxiv.org/html/2607.07993](https://arxiv.org/html/2607.07993)
Shiping Yang1,2Shining Liang2Weihao Liu3Wenbiao Ding2 Linjun Shou2Lu Cheng3Angel X\. Chang1 1Simon Fraser University2Microsoft3University of Illinois at Chicago

###### Abstract

Identifying faithfulness hallucinations in LLM\-generated outputs remains challenging due to the scarcity of high\-quality annotated data\. Recent work relies on advanced LLMs to synthesize training data, including rationales, labels, and hallucinated claims\. However, these methods treat the generator as a static component, limiting iterative improvement of the detector\. To address this limitation, we introduce Hallucination Self\-Play \(HSP\), a novel framework that enables the detector to bootstrap with an evolved generator\. HSP involves two roles initialized from the same base model, a detector that assesses the faithfulness of model outputs, and a generator that produces increasingly hard\-to\-detect hallucinated responses\. Specifically, the detector is first fine\-tuned on human\-labeled data and then employed as a reward model to train the generator via reinforcement learning from AI feedback \(RLAIF\)\. In turn, the evolved generator synthesizes hallucination data to further optimize the detector through rule\-based reinforcement learning\. Experiments on RAGTruth benchmark and two model families demonstrate that the proposed framework can progressively enhance a small LLM to match or even outperform advanced LLMs without external supervision\. Our code is available at[https://anonymous\.4open\.science/r/Hallucination\-Self\-Play\-50B5](https://anonymous.4open.science/r/Hallucination-Self-Play-50B5)\.

## 1Introduction

Despite the remarkable capabilities of large language models \(LLMs\) across diverse domains\(Liet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib35); Yuet al\.,[2025b](https://arxiv.org/html/2607.07993#bib.bib36); Zhanget al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib37); Jianget al\.,[2026](https://arxiv.org/html/2607.07993#bib.bib38)\), they remain prone to hallucinate when handling long\-tail knowledge or outdated information\. Retrieval\-augmented generation \(RAG\) has emerged as an effective paradigm to improve the factuality of model responses by grounding them in retrieved documents\(Arslanet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib31); Yanget al\.,[2025b](https://arxiv.org/html/2607.07993#bib.bib32)\)\. However, even with RAG, LLMs still suffer from faithfulness hallucinations, i\.e\., generating claims that are contradictory to or unsupported by the provided context\(Yanget al\.,[2023](https://arxiv.org/html/2607.07993#bib.bib2); Huanget al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib33); Jiang and Ferraro,[2026](https://arxiv.org/html/2607.07993#bib.bib39)\)\. Therefore, detecting such hallucinations is critical for providing trustworthy LLM services\.

Prior work leverages advanced LLMs to determine whether a model response contains hallucinations\(Dhuliawalaet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib26); Jacoviet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib25); Seoet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib6)\)\. While these methods achieve impressive performance, they are impractical for real\-world application due to high inference cost and latency\. This has motivated the development of lightweight and specialized detectors for efficient hallucination detection\. However, the high cost and scarcity of human annotation limit further performance scaling of detectors\. To address this, recent studies directly synthesize hallucinated claims using tailored generation pipelines\(Caoet al\.,[2023](https://arxiv.org/html/2607.07993#bib.bib27); Tanget al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib3); Tanet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib34); Leiet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib4)\)\. A key limitation of such approaches is that the hallucination generators are typically static, lacking adaptivity to the evolving capabilities of the detector\. As a result, detectors quickly reach a performance plateau, as the synthetic hallucinations become too easy to provide effective training signals for further improvement\.

To overcome this limitation, we proposeHallucination Self\-Play \(HSP\), a closed\-loop interaction between two roles: ageneratorand adetector, both initialized from the same base model\. The generator is optimized to produce diverse and challenging hallucinations based on the detector’s feedback, while the detector is trained via RLVR on the resulting synthetic data\. This interaction enables both roles to co\-evolve without external supervision\.

While self\-play has achieved great success in large language models for easy\-to\-verify tasks such as code generation and mathematical reasoning\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib14); Chenet al\.,[2025b](https://arxiv.org/html/2607.07993#bib.bib23); Lianget al\.,[2026](https://arxiv.org/html/2607.07993#bib.bib40)\), its application to hallucination detection remains underexplored due to the fundamental challenge of verifying the validity and correctness of generated hallucinations\. Without a reliable verification mechanism, the training process becomes vulnerable to reward hacking, where the generator may exploit superficial shortcuts to obtain rewards rather than producing genuinely challenging hallucinations\. To this end, we utilize the ground truth labels from QA dataset as a proxy for verification and introduce additional safeguards\.

We evaluate HSP on the RAGTruth benchmark under two settings\. In theDetector w/o CoTsetting, we show that HSP serves as a plug\-and\-play method that further improves a fine\-tuned detector, even with only a single round\. In the more challengingDetector w/ CoTsetting, we demonstrate that self\-play can enable a small model to achieve performance comparable to advanced LLMs in a fully self\-bootstrapping manner, without external rationale supervision\.

Our contribution can be summarized as three folds: \(1\) We extend self\-play paradigm to hallucination detection task, overcoming the limitation of static generators\. \(2\) We introduce additional verification mechanisms to mitigate reward hacking\. Further ablation studies confirm their effectiveness in suppressing such behavior, which is critical for maintaining synthetic data quality and stable detector training\. \(3\) Experiments demonstrate that our framework achieves strong performance and unlocks the potential of fully self\-bootstrapped learning\.

## 2Related Work

### 2\.1Hallucination Detection\.

Existing approaches to faithfulness hallucination detection mainly follow two directions\. The first relies on advanced LLM to evaluate LLM\-generated outputs\(Dhuliawalaet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib26); Jacoviet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib25); Seoet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib6)\)\. Although effective, these methods are often inefficient in practice, as they depend on advanced LLMs\.

To reduce cost, a second line of work focuses on training lightweight and deployable detectors\. Due to the scarcity of human\-annotated data, recent studies have turned to synthetic hallucination generation to train more capable detection models\(Caoet al\.,[2023](https://arxiv.org/html/2607.07993#bib.bib27)\)\. MiniCheck\(Tanget al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib3)\)synthesizes training data using advanced LLMs, while FactCG\(Leiet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib4)\)further increases data complexity via graph\-based multi\-hop augmentation\.

However, existing methods rely on static generators, whose fixed hallucination patterns gradually become easy for the detector and limit further improvement\. To address this, we propose a hallucination self\-play framework that bootstraps the detector with an evolving generator\.

### 2\.2Self\-Play\.

Self\-play is a paradigm in reinforcement learning, where an agent improves by interacting with copies of itself or co\-evolving counterparts\(Schmidhuber,[2013](https://arxiv.org/html/2607.07993#bib.bib18); Schaul,[2024](https://arxiv.org/html/2607.07993#bib.bib19)\)\. This paradigm became popular following the significant advances of AlphaGo\(Silveret al\.,[2017a](https://arxiv.org/html/2607.07993#bib.bib20)\)and AlphaZeroSilveret al\.\([2017b](https://arxiv.org/html/2607.07993#bib.bib21)\), which demonstrated that self\-play alone can yield superhuman performance in complex decision\-making tasks without human supervision\.

More recently, self\-play has been actively explored in the context of large language models\. Language Self\-Play\(Kubaet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib24)\)propose a Challenger\-Solver framework where language models improve themselves without human\-labeled data by generating training examples through self\-play\. AbsoluteZero\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib14)\)further explore this paradigm by achieving self\-play reasoning where the model generates reasoning problems and validates solutions through a code executor\. Another line of work formulates self\-play as an adversarial game\(Goodfellowet al\.,[2020](https://arxiv.org/html/2607.07993#bib.bib22)\)\. For example, SPC\(Chenet al\.,[2025b](https://arxiv.org/html/2607.07993#bib.bib23)\)designs an adversarial game between a generator and a critic to improve the critic’s capabilities\.

Despite these advances, most existing self\-play work focuses on improving reasoning performance on math or code tasks\. In contrast, our work applies self\-play to hallucination detection, introducing a Hallucination Self\-Play framework that enables the detector to continuously self\-improve\.

## 3Methodology

In this section, we proposeHallucinationSelf\-Play \(HSP\), a framework that bootstraps the detector by evolving a generator to automatically synthesize hard\-to\-detect hallucinated responses \(Figure[1](https://arxiv.org/html/2607.07993#S3.F1)\)\. We first formulate the tasks of the detector and the generator, and detail how these two roles are initialized from the same base model \(§[3\.1](https://arxiv.org/html/2607.07993#S3.SS1)\)\. Next, we introduce the training algorithms and the corresponding reward design for both roles \(§[3\.2](https://arxiv.org/html/2607.07993#S3.SS2)\)\. Finally, we describe the closed\-loop self\-play procedure, illustrating the iterative interaction between the detector and the generator \(§[3\.3](https://arxiv.org/html/2607.07993#S3.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2607.07993v1/x1.png)Figure 1:Overview of the HSP framework\. The generator is evolved via RLAIF, with the frozen detector providing reward signals\. Components for mitigating reward hacking are omitted for clarity \(Top\)\. The detector is optimized via RLVR on synthetic data produced by the frozen generators\.Faith Gendenotes the base modelℳ\\mathcal\{M\}prompted to generate faithful responses \(Bottom\)\. The two roles interact in a closed loop, with each role alternately frozen while the other is trained\.### 3\.1Two Roles: Detector and Generator

The HSP framework consists of two interacting roles: a detector and a generator\. Both roles are instantiated from the same base model but are specialized for different inputs, outputs, and training objectives\.

#### 3\.1\.1Task Formulation

##### Hallucination Detection\.

Given a grounding documentd​o​cdocand an LLM\-generated claimcc, the detector modelDDis responsible for determining whether the claim is faithful or hallucinated\. We consider a claim to be faithful if it is fully supported by the grounding document\. Conversely, a claim is regarded as hallucinated if it contradicts or cannot be verified using the provided context\. While previous work typically formulates hallucination detection as a binary classification task\(Yanget al\.,[2023](https://arxiv.org/html/2607.07993#bib.bib2); Tanget al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib3); Seoet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib6)\), many real\-world applications require locating the specific hallucinated spans\. To this end, followingNiuet al\.\([2024](https://arxiv.org/html/2607.07993#bib.bib1)\), we train the model to generate a structured output sequencezz, which represents hallucinated spans in a JSON format\. Formally, the detectorDDmodels the conditional probability:

PD​\(y,z∣d​o​c,c\),P\_\{D\}\(y,z\\mid doc,c\),\(1\)
where y is a binary label derived fromzz, withy=0y=0ifzzis empty \(i\.e\., no hallucinated spans are identified\), andy=1y=1otherwise\. Recent work suggests that learned reasoning process can facilitate hallucination span detection\(Suet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib5)\)\. Therefore, we further extend the detector to a variant that not only predicts hallucinated spans, but also generates a rationalec​o​tcot\(i\.e\., Chain\-of\-thought\) to justify its prediction\. This variant can be formulated as:

PD​\(y,z,c​o​t∣d​o​c,c\)\.P\_\{D\}\(y,z,cot\\mid doc,c\)\.\(2\)

##### Hallucination Generation\.

Given a queryqqand its corresponding grounding documentd​o​cdoc, the generator modelGGis designed to intentionally synthesize hallucinated claims\. Such synthetic claims are constructed to appear plausible yet contain inconsistencies with the given document\. Formally,

PG​\(c∣q,d​o​c,y=1\),P\_\{G\}\(c\\mid q,doc,y=1\),\(3\)
wherey=1y=1represents that the generator is conditioned to generate hallucination\.

#### 3\.1\.2Roles Initialization

This stage aims to equip the model with basic capabilities for hallucination detection and hallucination generation, serving as an initialization for the subsequent self\-play reinforcement learning stage\.

##### Initialize Detector\.

We cold\-start the detectorDDby training the base modelℳ\\mathcal\{M\}via supervised fine\-tuning \(SFT\) on a small dataset𝒟sft\\mathcal\{D\}\_\{\\text\{sft\}\}\. We consider two variants of the detector and describe how their SFT data is obtained\. For the detector variant without chain\-of\-thought,𝒟sft\\mathcal\{D\}\_\{\\text\{sft\}\}is sourced from the human\-annotated RAGTruth dataset\(Niuet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib1)\)\. Each training instance consists of a grounding documentd​o​cdoc, a generated claimcc, and a structured labelzzwith annotated hallucinated spans\.

For the detector variant with chain\-of\-thought, existing training datasets provide only prediction labels but lack rationale annotations\. A common approach to bridge this gap is to leverage advanced LLMs to synthesize data with rationales for distillation\(Songet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib7)\)\. In contrast, we investigate whether a small LLM can self\-bootstrap its reasoning capabilities for hallucination span detection in the absence of annotated rationales data\. Therefore, we employ a rejection sampling strategy to mine high\-quality reasoning paths from the model itself, using span\-level annotations as the filtering signal\. Specifically, for each\(d​o​c,c\)\(doc,c\)pair, we sample multiple candidate rationales from the base modelℳ\\mathcal\{M\}by instructing it to generate reasoning traces alongside hallucinated spans\. Candidates that fail to identify all labeled hallucination spans are rejected, and only those whose predicted spans fully cover the ground truth are retained\. The resulting samples are treated as pseudo\-labeled rationale data and aggregated to form𝒟sft\\mathcal\{D\}\_\{\\text\{sft\}\}\. Given the constructed SFT dataset, we finetune the detector to maximize the conditional likelihood of the target outputoo:

ℒsft​\(θD\)=−𝔼\(d​o​c,c,o\)∼𝒟sft​\[log⁡PD​\(o∣d​o​c,c\)\],\\mathcal\{L\}\_\{\\text\{sft\}\}\(\\theta\_\{D\}\)=\-\\mathbb\{E\}\_\{\(doc,c,o\)\\sim\\mathcal\{D\}\_\{\\text\{sft\}\}\}\\left\[\\log P\_\{D\}\(o\\mid doc,c\)\\right\],\(4\)
whereo=zo=zfor the detector without CoT, ando=\(z,c​o​t\)o=\(z,cot\)for the detector with CoT\.

##### Initialize Generator\.

We initialize the generatorGGfrom the same base modelℳ\\mathcal\{M\}using a prompt\-based approach\. Previous work often relies on predefined templates or heuristic rules to synthesize hallucinations, while we do not impose any fixed hallucination strategy in the prompt template\. This initialization encourages diverse hallucination patterns to emerge dynamically through interaction with the detector\. The prompt used for the generator is shown in Figure[2](https://arxiv.org/html/2607.07993#A2.F2)\.

### 3\.2Reinforcement Learning Training

While SFT provides a strong initialization for the detector, it is ill\-suited for subsequent self\-play training\. For the detector, annotated hallucination spans and reasoning paths are difficult to obtain from the generator\. Moreover, SFT tends to encourage mode covering, which can harm generalization when trained on ”adversarial” samples produced by the generator, whereas reinforcement learning favors mode\-seeking behavior\(Chenet al\.,[2025a](https://arxiv.org/html/2607.07993#bib.bib8)\)\. For the generator, the objective of synthesizing hard\-to\-detect hallucinations cannot be explicitly supervised, as the generator only receives scalar feedback from the detector\. Therefore, we adopt reinforcement learning for training both roles in hallucination self\-play\.

#### 3\.2\.1Policy Optimization Algorithm

Policy optimization methods for reinforcement learning of LLMs, such as PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.07993#bib.bib9)\)and GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib10)\), have been well studied\. Given the advantage of GRPO, including the removal of the critic model, we utilize GRPO to optimize both roles in our HSP framework\. Under this unified optimization algorithm, the two roles differ only in their input\-output formats and reward functions\.

For each training instancexx, we sample a group ofKKcandidate outputs\{o1,…,ok\}\\\{o\_\{1\},\\ldots,o\_\{k\}\\\}from the old policyℳold\\mathcal\{M\}\_\{\\text\{old\}\}\. Each candidate is then evaluated by a role\-specific reward function, producing reward\{r1,…,rk\}\\\{r\_\{1\},\\ldots,r\_\{k\}\\\}\. GRPO estimates an advantageAiA\_\{i\}for each output using relative rewards within the group, guiding policy updates according to the following objective:

ℒGRPO​\(ℳθ\)=𝔼x,\{oi\}∼ℳold​\[1K​∑i=1Kmin⁡\(wi​Ai,clip​\(wi,1−ϵ,1\+ϵ\)​Ai\)−β​𝔻KL​\(ℳθ∥ℳref\)\],\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\mathcal\{M\}\_\{\\theta\}\)=\\mathbb\{E\}\_\{x,\\\{o\_\{i\}\\\}\\sim\\mathcal\{M\}\_\{\\text\{old\}\}\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\min\\\!\\left\(w\_\{i\}A\_\{i\},\\;\\mathrm\{clip\}\(w\_\{i\},1\-\\epsilon,1\+\\epsilon\)\\,A\_\{i\}\\right\)\-\\beta\\,\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\\\!\\left\(\\mathcal\{M\}\_\{\\theta\}\\,\\\|\\,\\mathcal\{M\}\_\{\\text\{ref\}\}\\right\)\\right\],\(5\)
wherewi=ℳθ​\(oi∣x\)ℳold​\(oi∣x\)w\_\{i\}=\\frac\{\\mathcal\{M\}\_\{\\theta\}\(o\_\{i\}\\mid x\)\}\{\\mathcal\{M\}\_\{\\text\{old\}\}\(o\_\{i\}\\mid x\)\}\. Here,ℳref\\mathcal\{M\}\_\{\\text\{ref\}\}denotes the reference policy \(i\.e\., the initialized model\),ℳold\\mathcal\{M\}\_\{\\text\{old\}\}is the policy before the update,ϵ\\epsiloncontrols the clipping range, andβ\\betais the coefficient for KL regularization\.

#### 3\.2\.2Evolving Generator via RLAIF

Previous work has shown that task difficulty is crucial for effective reinforcement learning\(Zenget al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib12); Yuet al\.,[2025a](https://arxiv.org/html/2607.07993#bib.bib13)\)\. To enable the continuous improvement of the detector, we evolve the generator to produce hard\-to\-detect hallucinations using feedback from the detector, i\.e\., reinforcement learning from AI feedback \(RLAIF\)\.

##### Detector\-Guided Reward\.

Inspired byZhaoet al\.\([2025](https://arxiv.org/html/2607.07993#bib.bib14)\), we design a detector\-guided reward that encourages the generation of hard\-but\-solvable hallucinations\. Specifically, we rollout the detectorKKtimes with a high sampling temperature111In our experiments, we sample1010times with temperature1\.01\.0\.and estimate the average success rater^acc\\hat\{r\}\_\{\\text\{acc\}\}as a proxy for the learnability of a synthetic hallucination\. This reward can be defined as:

rdetector\-guided=\{0,if​r^acc=0,1−r^acc,otherwise,r\_\{\\text\{detector\-guided\}\}=\\begin\{cases\}0,&\\text\{if \}\\hat\{r\}\_\{\\text\{acc\}\}=0,\\\\ 1\-\\hat\{r\}\_\{\\text\{acc\}\},&\\text\{otherwise\},\\end\{cases\}\(6\)
wherer^acc=1K​∑i=1Kracc\(i\)\\hat\{r\}\_\{\\text\{acc\}\}=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}r\_\{\\text\{acc\}\}^\{\(i\)\}, andracc\(i\)r\_\{\\text\{acc\}\}^\{\(i\)\}is a binary indicator of whether the detector correctly identifies the synthetic hallucination in theii\-th rollout\.

##### Reward Hacking\.

When the generator is optimized solely with the reward defined in Eq\.[6](https://arxiv.org/html/2607.07993#S3.E6), the training objective is highly vulnerable to reward hacking\. Specifically, the generator can easily finds a shortcut strategies that maximize the expectation of reward without actually learning to produce high\-quality hallucinations\. A common failure mode is that the generator outputs a faithful answer, which the detector consequently fail to identify as hallucinated with high accuracy, thereby yielding a high reward\. This hacking behavior introduces incorrect labels into the synthetic data, causing the detector to receive noisy reward and fail to learn robust decision boundaries\.

##### Reward Gating Criteria\.

To address the reward hacking issue, we introduce two reward gating criteria to verify whether the generated response is indeed hallucinated\. These criteria are motivated by the definition of faithfulness hallucination\. A response that satisfies at least one criterion is considered hallucinated and receives a reward, while a response that fails all criteria receives a penalty\. This design prevents the generator from gaming the reward function by producing non\-hallucinated responses to obtain high rewards\.

First, a generated claim is gated as eligible for detector\-guided reward if it contradicts the provided context\. We use the ground\-truth answer from a QA dataset as a simplified indicator to assess whether the generated response conflicts with the given context\. Specifically, the response is deemed inconsistent with the grounding document if the correct answer is absent, and thus passes this reward gate\. In addition to a basic string\-matching check, we implement a model\-based method that instructs an LLM to identify whether the generated response contains aliases or variants of the answer\. The prompt can be found in Figure[3](https://arxiv.org/html/2607.07993#A2.F3)\.

Second, a generated claim is also gated as eligible for detector\-guided reward if it introduces facts that are unsupported by the grounding documents\. To implement this criterion, we employ a named entity recognition \(NER\) model222We use the NER modelen\_core\_web\_lgprovided by spaCy\.to extract entities from the query, the grounding documents, and the generated response\. If the response introduces entities that do not appear in either the query or the given context, it passes this reward gate as containing unsupported facts\.

##### Trivial Answer Penalty\.

Even with the above gating criteria, we observe an unexpected form of reward hacking: the generator learns to produce trivial or meaningless response, or to refuse answering\. Such responses are neither hallucinated nor correct, thereby bypass the reward gating criteria\. To address this issue, we introduce a hybrid detection strategy: refusal responses are identified via keyword\-based heuristics, while trivial or evasive outputs are detected with a model\-based approach333We use the base modelℳ\\mathcal\{M\}for all model\-based implementations to avoid introducing any external supervision\. The prompt template is provided in Figure[4](https://arxiv.org/html/2607.07993#A2.F4)\.\. All identified responses are penalized to suppress this behavior\.

##### Overall Reward\.

We integrate thedetector\-guided reward,reward gating criteria, andtrivial answer penaltyinto an overall reward for optimizing the generator\. The generator reward is formulated as:

rG=\{−1,if the response is trivial,𝕀gate⋅rdetector\-guided,otherwise,r\_\{G\}=\\begin\{cases\}\-1,&\\text\{if the response is trivial\},\\\\ \\mathbb\{I\}\_\{\\text\{gate\}\}\\cdot r\_\{\\text\{detector\-guided\}\},&\\text\{otherwise\},\\end\{cases\}\(7\)
where𝕀gate∈\{−1,1\}\\mathbb\{I\}\_\{\\text\{gate\}\}\\in\\\{\-1,1\\\}equals11if the generated response satisfies at least one gating criteria, and−1\-1otherwise\. This design imposes hard regularization on the generator, constraining the feasible solution space and restricting non\-zero rewards to valid hallucinated responses\. As a result, the generator is encouraged to synthesize hard yet learnable hallucinations that provide effective training signals for the detector\.

#### 3\.2\.3Optimizing Detector via RLVR

Reinforcement Learning with Verifiable Rewards \(RLVR\) has recently achieved great success in enhancing the reasoning capabilities of LLMs\(Guoet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib11)\)\. Recent research further demonstrates its effectiveness in hallucination span detection\(Suet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib5)\)\. Hence, we apply RLVR to optimize the detector in our HSP framework\.

##### Prediction Correctness Reward\.

Suet al\.\([2025](https://arxiv.org/html/2607.07993#bib.bib5)\)employ a span\-level reward function that computes the F1 score between predicted hallucination spans and ground\-truth annotations\. However, in our self\-play setting, the generator is typically a small LLM\. Such models struggle to produce hallucinated claims along with correct span annotations due to limited capabilities\. To ensure the data quality, we formulate the generator to produce hallucinated claims only\. Accordingly, we adopt a binary prediction correctness reward instead of a span\-level reward to optimize the detector\. Formally, given an input pair\(d​o​c,c\)\(doc,c\), the detector derives a binary predictiony^∈\{0,1\}\\hat\{y\}\\in\\\{0,1\\\}, wherey^=0\\hat\{y\}=0indicates a faithful claim andy^=1\\hat\{y\}=1indicates a hallucinated claim\. The detector reward is defined as:

rdetector=𝕀​\[y^=ygt\],r\_\{\\text\{detector\}\}=\\mathbb\{I\}\\big\[\\hat\{y\}=y\_\{\\text\{gt\}\}\\big\],\(8\)whereygt∈\{0,1\}y\_\{\\text\{gt\}\}\\in\\\{0,1\\\}is the ground\-truth label and𝕀​\[⋅\]\\mathbb\{I\}\[\\cdot\]is the indicator function\. Through verifiable reward signals, RLVR encourages the detector to refine its reasoning process and prediction strategy in a self\-corrective fashion\.

##### Dataset with Verifiable Labels\.

All hallucination responses synthesized by the generator are automatically labeled as hallucinated\. To construct balanced training data with faithful labels, we prompt the base modelℳ\\mathcal\{M\}to produce answers that strictly adhere to the provided grounding documents\. Although the base model is not guaranteed to be hallucination\-free, we empirically observe that the residual noise becomes acceptable after filtering against the ground\-truth answers, resulting in labels that are effective for detector training\. Importantly, we do not directly use the ground\-truth answers included in QA datasets for detector training\. These answers are typically very short and therefore deviate substantially from the distribution of model\-generated response\.

### 3\.3Hallucination Self\-Play Loop

We define hallucination self\-play as a closed\-loop interaction between a generator and a detector, in which learning signals are derived from their internal interaction rather than external supervision\. The generator is encouraged to produce hallucinated claims that challenge the target detector, while the detector aims to correctly identify the generated hallucinations\. This interaction yields dual learning signals: detector feedback for updating the generator and data with verifiable label for training the detector\. Notably, the two roles do not exchange parameters or gradients; instead, learning is driven solely by generated data and derived rewards, forming a stable closed\-loop self\-play process\.

##### Information Flow in the Self\-Play Loop\.

The self\-play training loop is driven by the flow of information between the two roles\. Given a query\-document pair, the generator synthesizes hallucinated claims, which are then evaluated by a frozen target detector\. The detector’s predictions are processed into a scalar reward signal and returned to the generator as feedback\. After the generator is evolved via RLAIF, we freeze the updated generator and use it to produce hallucination candidates for detector training\. To improve training efficiency, hallucinations are scored by Eq\.[7](https://arxiv.org/html/2607.07993#S3.E7)using the previous detector to mine learnable examples\. These selected samples are further combined with non\-hallucinated responses generated by the base model, forming a balanced training dataset\. The detector then predicts on this synthetic dataset and is optimized via RLVR using prediction correctness rewards\.

##### Dynamic Curriculum via Multi\-Round Self\-Play\.

Even after a single round of self\-play training, hallucinations produced by the updated generator quickly saturate and provide little additional learning signal for the detector\. As the detector improves, previously challenging hallucinations become easy, limiting further progress\. In contrast, multi\-round self\-play induces a dynamic curriculum that continuously adapts the difficulty of synthetic hallucinations to the detector’s evolving capability\. This dynamic adjustment of task difficulty maximizes learning efficiency and enables the generator and detector to iteratively co\-evolve under an evolving curriculum, driving continuous improvement without external supervision\.

## 4Experiments

Table 1:Hallucination detection performance on RAGTruth across three tasks\. We report response\-level precision, recall, and F1\. ForDetector w/ CoT, all experiments are conducted usingQwen2\.5\-7B\-Instructas the base model\.RSFTdenotes supervised fine\-tuning on CoT data mined via rejection sampling, whileRLVRfurther optimizes the RSFT model via RL using hallucination annotations from RAGTruth, without access to labeled rationales\.HSPmodels in this setting are trained iteratively, where each iteration continues training from the model obtained in the last round, forming a multi\-round hallucination self\-play\. ForFinetuned Models, entries marked with†\{\\dagger\}indicate models obtained by applying HSP training on top of the corresponding SFT checkpoint\. We highlight the best results within each setting in bold\.In this section, we conduct a series of experiments and analyses to demonstrate the effectiveness of our HSP framework\.

### 4\.1Experiment Setup

##### Evaluation\.

We use RAGTruth as evaluation benchmark, which includes three representative tasks in the retrieval\-augmented generation setting:Question Answering\(QA\),Data\-to\-Text, andSummarization\. Each task contains paired source documents, model\-generated responses, and human\-labeled hallucination spans\. We follow the RAGTruth evaluation protocol and report response\-level recall, precision, and f1 scores\. For theDetector w/ CoTsetting, we use the prompt template shown in Figure[5](https://arxiv.org/html/2607.07993#A2.F5)\.

##### Baselines\.

We compare several baselines, including\(1\) State\-of\-the\-Art LLMs: the proprietary model GPT\-4o\(Hurstet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib16)\)and the advanced open\-source model DeepSeek\-V3\.2\(Liuet al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib15)\);\(2\) Reasoning Models: Qwen3\-32B and Qwen3\-14B, which are optimized for long\-cot reasoning\(Yanget al\.,[2025a](https://arxiv.org/html/2607.07993#bib.bib17)\);\(3\) Supervised Fine\-Tuning \(SFT\): the base model trained on the RAGTruth dataset using standard supervised learning;\(4\) RAG\-HAT\(Songet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib7)\): a specialized hallucination detection model that outputs binary predictions with detailed explanations, trained on RAGTruth and a curated dataset containing rationales distilled from GPT\-4\-turbo\.

##### Implementation Details\.

We useQwen2\.5\-7B\-InstructandLlama\-3\.1\-8B\-Insructas the base model in our main results\. For theDetector w/o CoTsetting, we initialize the detector by fine\-tuning the base model on RAGTruth\(Niuet al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib1)\)\. For theDetector w/ CoTsetting, we instead cold\-start the detector using a small set of CoT data mined fromQwen2\.5\-7B\-Instructusing rejection sampling\. During self\-play training,HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2607.07993#bib.bib28)\)serves as the seed dataset for synthetic data generation\. Additional implementation details and hyperparameters can be found in the appendix[A](https://arxiv.org/html/2607.07993#A1)\.

### 4\.2Main Results

##### Detector without CoT\.

We first evaluate HSP in theDetector w/o CoTsetting by applying one round of self\-play on top of SFT checkpoints\. As shown in theFinetuned Modelssection of Table[1](https://arxiv.org/html/2607.07993#S4.T1), HSP consistently improves F1 over the corresponding SFT baselines for bothQwen2\.5\-7B\-InstructandLlama\-3\.1\-8B\-Instructacross all three tasks\. These results demonstrate that HSP serves as a model\-agnostic, plug\-and\-play post\-training strategy\. The improvements are primarily driven by higher recall while maintaining comparable precision, suggesting that self\-play enables detectors to identify a broader range of hallucinations without substantially increasing false positives\.

##### Detector with CoT\.

We next consider the more challengingDetector w/ CoTsetting, where neither human\-annotated rationales nor external LLM distillation data are available\. HSP demonstrates sustained improvements across self\-play rounds, with performance increasing from 72\.9 \(Round 1\) to 74\.3 \(Round 3\), reflecting an evolving self\-play curriculum\. Although trained solely on QA data, the reasoning capabilities optimized by RLVR show strong cross\-task generalization, boosting F1 on the Data\-to\-Text task from 72\.2 to 81\.4 \(\+9\.2 points\)\. Notably, we train a 7B model entirely through self\-play without any external rationale supervision, achieving performance comparable to the frontier proprietary modelGPT\-4o w/ CoT\(74\.3 vs\. 74\.5\)\. These results show that a compact model can fully self\-bootstrap strong hallucination detection capabilities under our framework\.

### 4\.3Ablation Study

We ablate the reward gating criteria and trivial answer penalty introduced in §[3\.2](https://arxiv.org/html/2607.07993#S3.SS2)to assess their impact on synthetic data quality\. All experiments useQwen3\-4B\-Instruct\-2507as the generator and are evaluated on 500 held\-out samples\. For these samples, we report the number of positive\-reward instances retained for detector training and their corresponding average reward\. To further evaluate data quality, we randomly sample 10 examples and manually assess the hallucination rate of the generated responses, defined as the fraction of responses that are genuinely hallucinated\.

As shown in Table[3](https://arxiv.org/html/2607.07993#A2.T3), removing the reward gating criteria causes the generator to collapse into producing non\-hallucinated responses, achieving a high reward of 1\.0 while the actual hallucination rate drops to 0%\. In contrast, the full HSP configuration maintains a high hallucination rate of 90%, at the cost of lower average reward and fewer retained samples\. These results confirm that both components are essential for suppressing reward hacking and ensuring that the synthetic training data remains informative for the detector\.

## 5Conclusion

In this work, we introduce Hallucination Self\-Play \(HSP\), a framework that enables hallucination detectors to self\-bootstrap without relying on external supervision\. HSP are defined as a closed\-loop interaction between a generator and a detector derived from the same base model\. The generator is continuously evolved via RLAIF to synthesize challenging yet learnable hallucinations, overcoming the limitations of static generators\. The detector is optimized via RLVR on the resulting synthetic data\. Experiments on RAGTruth demonstrate the effectiveness of HSP across diverse settings\. Overall, HSP demonstrates that self\-play offers an effective and scalable paradigm for continuously improving hallucination detectors\.

## Ethics Statement

While our method is designed to improve hallucination detection, the generator component of our HSP framework explicitly learns to produce hard\-to\-detect hallucinations during training\. Such a generator could potentially be misused to generate unfaithful content that is difficult to identify\. We emphasize that the generator in our framework is not intended for standalone deployment, but is used exclusively as a controlled component within a hallucination self\-play loop\.

## References

- M\. Arslan, H\. Ghanem, S\. Munawar, and C\. Cruz \(2024\)A survey on rag with llms\.Procedia computer science246,pp\. 3781–3790\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- Z\. Cao, Y\. Yang, and H\. Zhao \(2023\)Autohall: automated hallucination dataset generation for large language models\.arXiv preprint arXiv:2310\.00259\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07993#S2.SS1.p2.1)\.
- H\. Chen, N\. Razin, K\. Narasimhan, and D\. Chen \(2025a\)Retaining by doing: the role of on\-policy data in mitigating forgetting\.arXiv preprint arXiv:2510\.18874\.Cited by:[§3\.2](https://arxiv.org/html/2607.07993#S3.SS2.p1.1)\.
- J\. Chen, B\. Zhang, R\. Ma, P\. Wang, X\. Liang, Z\. Tu, X\. Li, and K\. K\. Wong \(2025b\)Spc: evolving self\-play critic via adversarial games for llm reasoning\.arXiv preprint arXiv:2504\.19162\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p2.1)\.
- S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. Weston \(2024\)Chain\-of\-verification reduces hallucination in large language models\.InFindings of the association for computational linguistics: ACL 2024,pp\. 3563–3578\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07993#S2.SS1.p1.1)\.
- I\. Goodfellow, J\. Pouget\-Abadie, M\. Mirza, B\. Xu, D\. Warde\-Farley, S\. Ozair, A\. Courville, and Y\. Bengio \(2020\)Generative adversarial networks\.Communications of the ACM63\(11\),pp\. 139–144\.Cited by:[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§3\.2\.3](https://arxiv.org/html/2607.07993#S3.SS2.SSS3.p1.1)\.
- L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§4\.1](https://arxiv.org/html/2607.07993#S4.SS1.SSS0.Px2.p1.1)\.
- A\. Jacovi, A\. Wang, C\. Alberti, C\. Tao, J\. Lipovetz, K\. Olszewska, L\. Haas, M\. Liu, N\. Keating, A\. Bloniarz,et al\.\(2025\)The facts grounding leaderboard: benchmarking llms’ ability to ground responses to long\-form input\.arXiv preprint arXiv:2501\.03200\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07993#S2.SS1.p1.1)\.
- Y\. Jiang and F\. Ferraro \(2026\)Beyond math: stories as a testbed for memorization\-constrained reasoning in llms\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5590–5607\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- Y\. Jiang, D\. Li, and F\. Ferraro \(2026\)DRP: distilled reasoning pruning with skill\-aware step decomposition for efficient large reasoning models\.External Links:2505\.13975,[Link](https://arxiv.org/abs/2505.13975)Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- J\. G\. Kuba, M\. Gu, Q\. Ma, Y\. Tian, V\. Mohan, and J\. Chen \(2025\)Language self\-play for data\-free training\.arXiv preprint arXiv:2509\.07414\.Cited by:[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p2.1)\.
- D\. Lei, Y\. Li, S\. Li, M\. Hu, R\. Xu, K\. Archer, M\. Wang, E\. Ching, and A\. Deng \(2025\)FactCG: enhancing fact checkers with graph\-based multi\-hop data\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5002–5020\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07993#S2.SS1.p2.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu,et al\.\(2025\)From generation to judgment: opportunities and challenges of llm\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 2757–2791\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- X\. Liang, Z\. Li, Y\. Gong, Y\. Wang, H\. Zhang, Y\. N\. Wu, W\. Chen,et al\.\(2026\)Sws: self\-aware weakness\-driven problem synthesis in reinforcement learning for llm reasoning\.Advances in Neural Information Processing Systems38,pp\. 56801–56839\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p4.1)\.
- A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.\(2025\)Deepseek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[§4\.1](https://arxiv.org/html/2607.07993#S4.SS1.SSS0.Px2.p1.1)\.
- C\. Niu, Y\. Wu, J\. Zhu, S\. Xu, K\. Shum, R\. Zhong, J\. Song, and T\. Zhang \(2024\)Ragtruth: a hallucination corpus for developing trustworthy retrieval\-augmented language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10862–10878\.Cited by:[§3\.1\.1](https://arxiv.org/html/2607.07993#S3.SS1.SSS1.Px1.p1.5),[§3\.1\.2](https://arxiv.org/html/2607.07993#S3.SS1.SSS2.Px1.p1.7),[§4\.1](https://arxiv.org/html/2607.07993#S4.SS1.SSS0.Px3.p1.1)\.
- T\. Schaul \(2024\)Boundless socratic learning with language games\.arXiv preprint arXiv:2411\.16905\.Cited by:[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p1.1)\.
- J\. Schmidhuber \(2013\)Powerplay: training an increasingly general problem solver by continually searching for the simplest still unsolvable problem\.Frontiers in psychology4,pp\. 313\.Cited by:[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§3\.2\.1](https://arxiv.org/html/2607.07993#S3.SS2.SSS1.p1.1)\.
- W\. Seo, S\. Han, J\. Jung, B\. Newman, S\. Lim, S\. Lee, X\. Lu, Y\. Choi, and Y\. Yu \(2025\)Verifying the verifiers: unveiling pitfalls and potentials in fact verifiers\.arXiv preprint arXiv:2506\.13342\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07993#S2.SS1.p1.1),[§3\.1\.1](https://arxiv.org/html/2607.07993#S3.SS1.SSS1.Px1.p1.5)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3\.2\.1](https://arxiv.org/html/2607.07993#S3.SS2.SSS1.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)Hybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[Appendix A](https://arxiv.org/html/2607.07993#A1.p1.1)\.
- D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.\(2017a\)Mastering chess and shogi by self\-play with a general reinforcement learning algorithm\.arXiv preprint arXiv:1712\.01815\.Cited by:[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p1.1)\.
- D\. Silver, J\. Schrittwieser, K\. Simonyan, I\. Antonoglou, A\. Huang, A\. Guez, T\. Hubert, L\. Baker, M\. Lai, A\. Bolton,et al\.\(2017b\)Mastering the game of go without human knowledge\.nature550\(7676\),pp\. 354–359\.Cited by:[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p1.1)\.
- J\. Song, X\. Wang, J\. Zhu, Y\. Wu, X\. Cheng, R\. Zhong, and C\. Niu \(2024\)RAG\-hat: a hallucination\-aware tuning pipeline for llm in retrieval\-augmented generation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 1548–1558\.Cited by:[§3\.1\.2](https://arxiv.org/html/2607.07993#S3.SS1.SSS2.Px1.p2.4),[§4\.1](https://arxiv.org/html/2607.07993#S4.SS1.SSS0.Px2.p1.1)\.
- H\. Su, T\. Hu, H\. S\. Koppula, K\. Krishna, H\. Pouransari, C\. Hsieh, C\. Koc, J\. Y\. Cheng, O\. Tuzel, and R\. Vemulapalli \(2025\)Learning to reason for hallucination span detection\.arXiv preprint arXiv:2510\.02173\.Cited by:[§3\.1\.1](https://arxiv.org/html/2607.07993#S3.SS1.SSS1.Px1.p3.5),[§3\.2\.3](https://arxiv.org/html/2607.07993#S3.SS2.SSS3.Px1.p1.4),[§3\.2\.3](https://arxiv.org/html/2607.07993#S3.SS2.SSS3.p1.1)\.
- Z\. Tan, D\. Li, S\. Wang, A\. Beigi, B\. Jiang, A\. Bhattacharjee, M\. Karami, J\. Li, L\. Cheng, and H\. Liu \(2024\)Large language models for data annotation and synthesis: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 930–957\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1)\.
- L\. Tang, P\. Laban, and G\. Durrett \(2024\)MiniCheck: efficient fact\-checking of llms on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8818–8847\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07993#S2.SS1.p2.1),[§3\.1\.1](https://arxiv.org/html/2607.07993#S3.SS1.SSS1.Px1.p1.5)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2607.07993#S4.SS1.SSS0.Px2.p1.1)\.
- S\. Yang, R\. Sun, and X\. Wan \(2023\)A new benchmark and reverse validation method for passage\-level hallucination detection\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 3898–3908\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1),[§3\.1\.1](https://arxiv.org/html/2607.07993#S3.SS1.SSS1.Px1.p1.5)\.
- S\. Yang, J\. Wu, W\. Ding, N\. Wu, S\. Liang, M\. Gong, H\. Zhang, and D\. Zhang \(2025b\)Quantifying the robustness of retrieval\-augmented language models against spurious features in grounding data\.arXiv preprint arXiv:2503\.05587\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§4\.1](https://arxiv.org/html/2607.07993#S4.SS1.SSS0.Px3.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025a\)Dapo: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§3\.2\.2](https://arxiv.org/html/2607.07993#S3.SS2.SSS2.p1.1)\.
- Y\. Yu, Y\. Zhang, D\. Zhang, X\. Liang, H\. Zhang, X\. Zhang, M\. Khademi, H\. H\. Awadalla, J\. Wang, Y\. Yang,et al\.\(2025b\)Chain\-of\-reasoning: towards unified mathematical reasoning in large language models via a multi\-paradigm perspective\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 24914–24937\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- W\. Zeng, Y\. Huang, Q\. Liu, W\. Liu, K\. He, Z\. Ma, and J\. He \(2025\)Simplerl\-zoo: investigating and taming zero reinforcement learning for open base models in the wild\.arXiv preprint arXiv:2503\.18892\.Cited by:[§3\.2\.2](https://arxiv.org/html/2607.07993#S3.SS2.SSS2.p1.1)\.
- H\. Zhang, S\. Yang, X\. Liang, C\. Shang, Y\. Jiang, C\. Tao, J\. Xiong, H\. K\. So, R\. Xie, A\. X\. Chang,et al\.\(2025\)Find your optimal teacher: personalized data synthesis via router\-guided multi\-teacher distillation\.arXiv preprint arXiv:2510\.10925\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p1.1)\.
- A\. Zhao, Y\. Wu, Y\. Yue, T\. Wu, Q\. Xu, M\. Lin, S\. Wang, Q\. Wu, Z\. Zheng, and G\. Huang \(2025\)Absolute zero: reinforced self\-play reasoning with zero data\.arXiv preprint arXiv:2505\.03335\.Cited by:[§1](https://arxiv.org/html/2607.07993#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.07993#S2.SS2.p2.1),[§3\.2\.2](https://arxiv.org/html/2607.07993#S3.SS2.SSS2.Px1.p1.2)\.
- Y\. Zheng, R\. Zhang, J\. Zhang, Y\. Ye, Z\. Luo, Z\. Feng, and Y\. Ma \(2024\)LlamaFactory: unified efficient fine\-tuning of 100\+ language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),Bangkok, Thailand\.External Links:[Link](http://arxiv.org/abs/2403.13372)Cited by:[Appendix A](https://arxiv.org/html/2607.07993#A1.p1.1)\.

## Appendix AImplementation Details

All experiments are conducted on 8 NVIDIA A100 GPUs\. We implement SFT training usingLlamaFactory\(Zhenget al\.,[2024](https://arxiv.org/html/2607.07993#bib.bib30)\)and RL training using theverlframework\(Shenget al\.,[2025](https://arxiv.org/html/2607.07993#bib.bib29)\)\. Each training stage for both the detector and generator uses 10k samples\. The detailed hyperparameters for both training stages are summarized in Table[2](https://arxiv.org/html/2607.07993#A1.T2)\.

Table 2:Training hyperparameters for SFT and RL stages\.
## Appendix BPrompt Templates

Below is a question and related passages\. Your task is to write aplausible but incorrect answerthat appears confident and fluent, yetcontradicts or adds information not supported by the reference passages\.Please follow these rules:•Do not copy sentences directly from the passages\.•The answer should look natural and persuasive\.•The hallucinated information should be related to the topic, not random\.•The hallucination can appear as wrong facts, wrong reasoning, or unsupported claims\.Below is a question: \{question\}Below are related passages: \{reference\}Your hallucinated answer:Figure 2:Prompt template for hallucination generator\.Does the following response mention the same entity as the ground truth answer \(including abbreviations, aliases, and other variants\)?Response: \{response\}Ground Truth: \{ground truth\}Answer only YES or NO\.Figure 3:Prompt template for model\-based answer alias detection, used as part of the reward gating criteria\.You are given a question and a model\-generated answer\.Your task is to determine whether the answer is anEVASIVE ANSWER\.Definition:An evasive answer is one that:•Does NOT directly answer the key attribute\(s\) explicitly asked in the question, AND•Avoids providing the required information by giving vague descriptions, generic restatements, or partial information\.An evasive answer is NOT the same as a hallucinated answer\.•Hallucination introduces incorrect facts\.•Evasion avoids answering the question\.You should output:•“EVASIVE” if the answer fails to provide the required information\.•“NOT EVASIVE” if the answer directly answers the question\.Focus strictly on whether thequestion is answered, not on correctness of background details\.Question:Claude\-Auguste Lamy discovered the element thallium independently from this English chemist who died in what year?Answer:Claude\-Auguste Lamy discovered the element thallium independently from William Crookes, who was a prominent English chemist active in the mid\-19th century\.Explanation:The question explicitly asks for the year of death\. The answer identifies the chemist but does not provide the requested year\.Label: EVASIVEQuestion:Claude\-Auguste Lamy discovered the element thallium independently from this English chemist who died in what year?Answer:Claude\-Auguste Lamy discovered thallium independently from William Crookes, who died in 1905\.Explanation:The answer directly provides the requested information \(a year of death\), but the year is incorrect\. This is a hallucinated answer, not an evasive one\.Label: NOT EVASIVEQuestion:\{question\}Answer:\{response\}Explanation:Label:Figure 4:Prompt template for model\-based evasive answer detection, used as part of the trivial answer penalty\.### B\.1Detector Prompt Template

QABelow is a question:\{question\}Below are related passages:\{reference\}Below is an answer:\{response\}Your task is to determine whether the answer contains hallucinations\. First, provide reasoning with the following format:\#\# Step 1: < your first reasoning step \>
\#\# Step 2: < your next reasoning step \>
\.\.\.\(add as many steps as needed\)Then, compile the labeled hallucinated spans into a JSON dict, with a key"hallucination list"and its value is a list of hallucinated spans\. If there exist potential hallucinations:\{"hallucination list": \[span1, span2, \.\.\.\]\}\. Otherwise:\{"hallucination list": \[\]\}\.Output:SummarizationBelow is the original news:\{reference\}Below is a summary of the news:\{response\}Your task is to determine whether the summary contains hallucinations\. First, provide reasoning with the following format:\#\# Step 1: < your first reasoning step \>
\#\# Step 2: < your next reasoning step \>
\.\.\.\(add as many steps as needed\)Then, compile the labeled hallucinated spans into a JSON dict, with a key"hallucination list"and its value is a list of hallucinated spans\. If there exist potential hallucinations:\{"hallucination list": \[span1, span2, \.\.\.\]\}\. Otherwise:\{"hallucination list": \[\]\}\.Output:Data\-to\-TextBelow is a structured data in the JSON format:\{reference\}Below is an overview article written in accordance with the structured data:\{response\}Your task is to determine whether the article contains hallucinations\. First, provide reasoning with the following format:\#\# Step 1: < your first reasoning step \>
\#\# Step 2: < your next reasoning step \>
\.\.\.\(add as many steps as needed\)Then, compile the labeled hallucinated spans into a JSON dict, with a key"hallucination list"and its value is a list of hallucinated spans\. If there exist potential hallucinations:\{"hallucination list": \[span1, span2, \.\.\.\]\}\. Otherwise:\{"hallucination list": \[\]\}\.Output:Figure 5:Prompt templates used for the detector w/ CoT across three task types: QA, Summarization, and Data\-to\-Text\.Table 3:Ablation on reward hacking mitigation mechanisms\.Hallucination Ratemeasures the fraction of generated responses that are genuinely hallucinated\.Trainable Samplesdenotes the number of samples with positive reward retained for detector training\.