HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation

# HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation
Source: [https://arxiv.org/html/2610.03063](https://arxiv.org/html/2610.03063)
Yuxin JiangJinpeng LiShuning SunAffiliation:Fei Mi, Haoli Bai, Lifeng ShangAffiliation:Huawei Technologies

###### Abstract

Large Language Models \(LLMs\) are prone to generating hallucinated content, which compromises their reliability in knowledge\-intensive tasks\. To address this challenge without sacrificing creativity, we proposeHARPO, a reinforcement learning framework designed to jointly optimize faithfulness and creativity\. HARPO incorporates a Hallucination\-Aware Generative Reward Model \(HA\-GRM\), trained via verifiable feedback, to assess both faithfulness and writing quality\. A Selective Activation Mechanism \(SAM\) activates writing rewards only for outputs judged hallucination\-free by HA\-GRM, while a data curriculum progressively shifts training from creative writing to hallucination\-centric tasks\. On RAGTruth, our Qwen3\-4B\-based HA\-GRM achieves a response\-level F1 score of 78\.08%, compared with 66\.37% for the supervised fine\-tuning baseline\. Experiments on Qwen2\.5 and Qwen3 models from 1\.7B to 8B parameters show improvements in both faithful generation and writing quality\. On Qwen3\-4B, HARPO reduces the HA\-GRM\-judged hallucination rate on MultiHopRAG from 3\.29% to 1\.02%, while increasing the Arena\-Hard\-v2\.0 creative\-writing score from 16\.95% to 27\.54%\.

## 1Introduction

Large Language Models \(LLMs\) have demonstrated impressive capabilities in generating fluent, human\-like text\. However, their reliability remains a critical concern due to hallucinated outputs that are unfaithful with respect to a given source[Ji et al\. \(2023\)](https://arxiv.org/html/2610.03063#bib.bib1)\. Such inaccuracies undermine the trustworthiness and utility of LLMs, particularly in high\-stakes applications\.

While reinforcement learning \(RL\) techniques have been widely adopted to boost the performance of large language models \(LLMs\)[Wei et al\. \(2026\)](https://arxiv.org/html/2610.03063#bib.bib34), optimizing for a single capability often comes at the cost of degraded performance in other dimensions\. For instance, the RLMR framework[Liao et al\. \(2026\)](https://arxiv.org/html/2610.03063#bib.bib7)underscores the inherent challenge of preserving subjective textual quality while adhering to rigid objective constraints\. Similarly,[Damani et al\. \(2026\)](https://arxiv.org/html/2610.03063#bib.bib33)show that conventional RL can impair calibration on out\-of\-domain benchmarks and propose RLCR to jointly optimize correctness and calibration\.

In this paper, we address the conflict betweenhallucination mitigationandcreative generation\. Mitigating hallucinations demands precision and caution, whereas creative writing requires richness and novelty\. Existing methods struggle to balance these competing objectives: single\-reward strategies fail to improve both simultaneously, while naive data mixing often leads to interference between faithful adherence and expressive quality\.

To address this problem, we introduceHARPO\(Hallucination\-AwareReinforcement forPolicyOptimization\), a framework designed to jointly optimize faithfulness and creativity\. Prior work has established reinforcement learning with verifiable span\-level rewards for hallucination detection\([Su et al\., 2026](https://arxiv.org/html/2610.03063#bib.bib6)\)\. HARPO combines hallucination evaluation and writing preference evaluation in a Hallucination\-Aware Generative Reward Model \(HA\-GRM\), trained using Reinforcement Learning with Verifiable Rewards \(RLVR\)\([Shao et al\., 2024](https://arxiv.org/html/2610.03063#bib.bib5)\)\. We combine response\-level classification with an auxiliary reward based on hallucinated\-span prediction to supervise faithfulness evaluation\. By incorporating pairwise preference data for creative writing, HA\-GRM also learns to evaluate stylistic quality\.

To effectively utilize these signals during policy optimization, we propose a Selective Activation Mechanism \(SAM\)\. Unlike traditional fixed\-weight scalarization, prone to reward conflicts, SAM acts as a conditional gate: the writing reward is activated only if HA\-GRM judges the response to be hallucination\-free\. Finally, to stabilize the training process, we design a data curriculum strategy that gradually shifts the data distribution from creative writing instructions to hallucination\-focused scenarios\. This approach mitigates the catastrophic forgetting often observed in single\-objective alignment, preserving broad linguistic proficiency while sharpening evidential rigor\.

HA\-GRM achieves higher Overall response\-level F1 than the base model and SFT, while SFT remains stronger on QA detection and the base model retains a higher D2T F1\. We evaluate HARPO on Qwen2\.5 and Qwen3 models from 1\.7B to 8B parameters\. Relative to the base Qwen3\-4B model, HARPO reduces the HA\-GRM\-judged summarization hallucination rate by over 85% while increasing creative\-writing scores\. Compared with linear reward mixing on Qwen3\-4B, HARPO has a lower summarization hallucination rate and a higher General score, with a slightly higher MultiHopRAG hallucination rate and a lower creative\-writing score\. The curriculum comparison similarly shows a trade\-off between hallucination rates and writing performance\.

The contributions are as follows:

- •We introduce HARPO, a framework that combines hallucination\-aware reward modeling, conditional reward aggregation, and a creative\-to\-faithful data curriculum to jointly optimize faithful generation and writing quality\.
- •We train HA\-GRM with response\-level labels, span\-level annotations, and writing preferences, and use SAM to activate writing rewards for responses predicted to be hallucination\-free by HA\-GRM\.
- •Extensive experiments demonstrate that HARPO consistently enhances performance across various model families and scales, confirming its effectiveness\.

## 2Methodology

![Refer to caption](https://arxiv.org/html/2610.03063v1/training_frame_work.png)Figure 1:The HARPO Framework\.Left: TheData Curriculum Strategysmoothly shifts the training distribution from creative to faithful tasks via cosine annealing\. Right: TheSelective Activation Mechanismactivates writing rewards only for responses judged hallucination\-free by HA\-GRM\.We introduce HARPO, a framework designed to mitigate hallucinations in large language models \(LLMs\) while preserving creative generation capabilities\. Our approach integrates a Hallucination\-Aware Generative Reward Model \(HA\-GRM\) with a curriculum\-driven reinforcement learning strategy\. The optimization process features two core mechanisms: a Selective Activation Mechanism \(SAM\) that gates writing rewards based on HA\-GRM’s hallucination predictions, and a dynamic curriculum that progressively shifts the training distribution from creative to hallucination\-sensitive tasks\.

### 2\.1HA\-GRM: Reward Model Training

#### Task Formulation\.

We train the HA\-GRM to detect hallucinations at both the response and span levels, following the annotation structure of RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.03063#bib.bib13)\)\. Given the input contextccand the generated responsey=\(y1,y2,…,yT\)y=\(y\_\{1\},y\_\{2\},\\ldots,y\_\{T\}\)consisting ofTTcharacters, the model needs to first identify whether the responseyyis hallucinated and then locate all hallucinated spansSS, defined as portions of text inyythat lack support fromcc\. Additionally, to maintain expressive quality, we incorporate pairwise preference data, encouraging the model to evaluate responses based on helpfulness, relevance, conciseness, creativity, and completeness, similar to Arena\-Hard\([Li et al\., 2025](https://arxiv.org/html/2610.03063#bib.bib11)\)\.

#### Rule\-Based Reinforcement Learning\.

We fine\-tune the HA\-GRM using rule\-based online RL, adopting the GRPO setting\([Shao et al\., 2024](https://arxiv.org/html/2610.03063#bib.bib5)\)\. During rollout, the model predicts hallucinated spans and pointwise response\-level labels for hallucination detection, while generating pairwise comparisons for creative writing\. The reward signal combines format validity and prediction accuracy\.

For hallucination detection, we incorporate the span\-F1 metric alongside response\-level accuracy\. LetS^\\hat\{S\}be the predicted hallucination spans andSSbe the ground\-truth spans\. The hallucination rewardrhr\_\{h\}is

rh=\{1\+span\-F1​\(S,S^\),valid format & acc\.1,valid format only0,invalid formatr\_\{h\}=\\begin\{cases\}1\+\\text\{span\-F1\}\(S,\\hat\{S\}\),&\\text\{valid format \\& acc\.\}\\\\ 1,&\\text\{valid format only\}\\\\ 0,&\\text\{invalid format\}\\end\{cases\}\(1\)
For writing preference evaluation, the rewardrwr\_\{w\}is

rw=\{2,correct format & acc\.1,correct format only0,incorrect formatr\_\{w\}=\\begin\{cases\}2,&\\text\{correct format \\& acc\.\}\\\\ 1,&\\text\{correct format only\}\\\\ 0,&\\text\{incorrect format\}\\end\{cases\}\(2\)
These reward schemes are applied separately to the pointwise hallucination data and pairwise creative writing data\.

### 2\.2HARPO: Hallucination\-Aware Reinforcement for Policy Optimization

We now present the details of HARPO, a framework designed to balance the competing objectives of hallucination reduction and creative generation\. We employ reinforcement learning rather than direct Supervised Fine\-Tuning\. This approach enables us to optimize the model using unlabelled prompts without requiring ground\-truth reference texts\. To guide this process, we introduce two novel components: first, the Selective Activation Mechanism, which functions as the core scoring metric during optimization; and second, a data curriculum strategy that schedules the training distribution to smooth the transition between hallucination\-centric and creativity\-centric tasks\.

#### Reinforcement Learning with GRPO\.

As our reinforcement learning framework, we adopt Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.03063#bib.bib5)\), a variant of REINFORCE\([Williams, 1992](https://arxiv.org/html/2610.03063#bib.bib8)\)that utilizes group\-based advantage normalization\. To optimize the LLM policyπθ\\pi\_\{\\theta\}, given a promptxx, we sample a group ofKKrollouts\{τk\}k=1K\\\{\\tau\_\{k\}\\\}^\{K\}\_\{k=1\}from the current policyπθ\\pi\_\{\\theta\}\. Each rolloutτk\\tau\_\{k\}is assigned a reward scoreRk=R⁡\(τk\)R\_\{k\}=R\(\\tau\_\{k\}\)calculated by our reward system\. The policy is updated via the clipped policy gradient objective:

ℒGRPO\\displaystyle\\mathcal\{L\}\_\{\\text\{GRPO\}\}=1K∑k=1K1\|τk\|∑t=1\|τk\|min\[wk,t\(θ\)⋅A^k,t,\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\frac\{1\}\{\|\\tau\_\{k\}\|\}\\sum\_\{t=1\}^\{\|\\tau\_\{k\}\|\}\\min\\left\[w\_\{k,t\}\(\\theta\)\\cdot\\hat\{A\}\_\{k,t\},\\right\.clip\(wk,t\(θ\),1−ϵlow,1\+ϵhigh\)⋅A^k,t\]\\displaystyle\\quad\\left\.\\text\{clip\}\(w\_\{k,t\}\(\\theta\),1\-\\epsilon\_\{\\text\{low\}\},1\+\\epsilon\_\{\\text\{high\}\}\)\\cdot\\hat\{A\}\_\{k,t\}\\right\]−β⋅DKL\(πθ∥πref\)\.\\displaystyle\\quad\-\\beta\\cdot D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\parallel\\pi\_\{\\text\{ref\}\}\)\.\(3\)
where the token\-level importance weight is defined as the probability ratio:

wk,t​\(θ\)=πθ​\(τk,t\|x,τk,<t\)πθold​\(τk,t\|x,τk,<t\)\.\\displaystyle w\_\{k,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(\\tau\_\{k,t\}\|x,\\tau\_\{k,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\tau\_\{k,t\}\|x,\\tau\_\{k,<t\}\)\}\.\(4\)Here,τk,t\\tau\_\{k,t\}denotes the token generated at stepttin rolloutkk, andτk,<t\\tau\_\{k,<t\}denotes the preceding context\. The group\-normalized advantageA^k,t\\hat\{A\}\_\{k,t\}is computed using the rewards of theKKrollouts sampled for the same input:

A^k,t=R⁡\(τk\)−mean​\(\{Ri\}i=1K\)std​\(\{Ri\}i=1K\)\+δ\.\\displaystyle\\hat\{A\}\_\{k,t\}=\\frac\{R\(\\tau\_\{k\}\)\-\\text\{mean\}\(\\\{R\_\{i\}\\\}^\{K\}\_\{i=1\}\)\}\{\\text\{std\}\(\\\{R\_\{i\}\\\}^\{K\}\_\{i=1\}\)\+\\delta\}\.\(5\)whereδ\\deltais a small constant for numerical stability\. Following recent advances in RL for LLM reasoning such as DAPO\([Yu et al\., 2025](https://arxiv.org/html/2610.03063#bib.bib9)\), we adopt asymmetric clipping \(clip\-higher\) withϵlow<ϵhigh\\epsilon\_\{\\text\{low\}\}<\\epsilon\_\{\\text\{high\}\}\. This strategy encourages the exploration of novel token sequences while maintaining training stability\. The coefficientβ\\betascales the KL divergence termDKL\(πθ∥πref\)D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\parallel\\pi\_\{\\text\{ref\}\}\), ensuring the policy remains close to the reference modelπref\\pi\_\{\\text\{ref\}\}to prevent mode collapse\.

#### Selective Activation Mechanism \(SAM\)\.

A core challenge in aligning LLMs for both reliability and quality is the potential conflict between objectives\. Standard approaches typically employ a linear scalarization of rewards,Rtotal=w1​Rhallu\+w2​RcreatR\_\{\\text\{total\}\}=w\_\{1\}R\_\{\\text\{hallu\}\}\+w\_\{2\}R\_\{\\text\{creat\}\}\. However, this fixed\-weight strategy creates an optimization landscape where the model may learn to exploit the reward function\. We propose the Selective Activation Mechanism \(SAM\), a hierarchical reward aggregation strategy that activates the writing reward only when HA\-GRM predicts no hallucinated spans\.

Letxxdenote the input prompt andyydenote the generated response corresponding to a rolloutτk\\tau\_\{k\}\. Our HA\-GRM, described in Section[2\.1](https://arxiv.org/html/2610.03063#S2.SS1), provides two distinct scalar signals:

1. 1\.Hallucination Reward \(RhalluR\_\{\\text\{hallu\}\}\):A score based on HA\-GRM’s prediction of whether the responseyyis grounded in the context\. LetS^\\hat\{S\}denote the set of hallucinated spans predicted by HA\-GRM, and let\|⋅\|\|\\cdot\|denote the text length\. The hallucination reward is calculated as: Rhallu​\(x,y\)=\{1,if​S^=∅−\|S^\|\|y\|,if​S^≠∅R\_\{\\text\{hallu\}\}\(x,y\)=\\begin\{cases\}1,&\\text\{if \}\\hat\{S\}=\\emptyset\\\\ \-\\frac\{\|\\hat\{S\}\|\}\{\|y\|\},&\\text\{if \}\\hat\{S\}\\neq\\emptyset\\end\{cases\}\(6\)This formulation imposes a penalty proportional to the density of hallucinated content when errors occur\.
2. 2\.Writing Reward \(RcreatR\_\{\\text\{creat\}\}\):A score evaluating the helpfulness, relevance, conciseness, and novelty of the response\. This score is derived from a pairwise evaluation comparing the model’s outputyyagainst a reference answerrr\.

Under the SAM framework, the writing reward is gated by HA\-GRM’s hallucination prediction\. The final rewardRSAMR\_\{\\text\{SAM\}\}assigned to a rollout is defined as:

RSAM​\(x,y,r\)\\displaystyle R\_\{\\text\{SAM\}\}\(x,y,r\)=Rhallu​\(x,y\)\\displaystyle=R\_\{\\text\{hallu\}\}\(x,y\)\(7\)\+𝕀\[S^=∅\]⋅Rcreat\(x,y,r\)\.\\displaystyle\\quad\+\\mathbb\{I\}\[\\hat\{S\}=\\emptyset\]\\cdot R\_\{\\text\{creat\}\}\(x,y,r\)\.where𝕀⁡\[⋅\]\\mathbb\{I\}\[\\cdot\]is the indicator function andS^\\hat\{S\}is the set of hallucinated spans predicted by HA\-GRM\. The writing reward is activated whenS^=∅\\hat\{S\}=\\emptysetand suppressed otherwise\. This gate constrains reward assignment according to HA\-GRM’s prediction; it does not guarantee that the response is truly hallucination\-free\.

#### Data Curriculum Strategy\.

The distinct nature of our two types of training data, creative writing and hallucination mitigation, presents a significant optimization challenge\. Creative writing tasks demand open\-ended generation with diverse vocabulary, whereas hallucination mitigation requires strict adherence to evidence and factual constraints\. Directly mixing these datasets in a static ratio often leads to optimization interference\.

To address this, we design a dynamic data curriculum that modulates the training distribution over time\. We adopt aCreative→\\toFaithfulprogression strategy for two reasons: \(1\)Model Affinity:We initialize training with a higher proportion of creative writing data\. This leverages the pre\-trained model’s inherent linguistic capabilities, stabilizing the initial policy optimization before introducing strict factual constraints\. \(2\)Prevention of Catastrophic Forgetting:Rather than abruptly switching tasks, we employ a smooth decay schedule\. This ensures that even as the focus shifts toward hallucination mitigation, a minimum proportion of creative writing data is maintained to preserve language proficiency\.

Formally, letTTdenote the total number of training steps\. We definer⁡\(t\)r\(t\)as the sampling ratio of creative writing data at steptt\. We utilize a cosine annealing schedule to smoothly reduce this ratio from an initial high to a fixed minimum:

r⁡\(t\)=r0⋅cos⁡\(π2​T⋅t\)\+rmin\.\\displaystyle r\(t\)=r\_\{0\}\\cdot\\cos\\left\(\\frac\{\\pi\}\{2T\}\\cdot t\\right\)\+r\_\{\\min\}\.\(8\)
wherermin=0\.05r\_\{\\min\}=0\.05represents the minimum proportion of creative data maintained at the end of training to prevent forgetting, andr0=0\.9r\_\{0\}=0\.9represents the magnitude of the decay\. Under this schedule, the training begins with a creative data ratio ofr⁡\(0\)=r0\+rminr\(0\)=r\_\{0\}\+r\_\{\\min\}and gradually converges tor⁡\(T\)=rminr\(T\)=r\_\{\\min\}as the model becomes more robust to hallucination\-sensitive tasks\.

## 3Experiments on HA\-GRM

Model NameRAGTruth \(Summ\)RAGTruth \(QA\)RAGTruth \(D2T\)OverallPRF1PRF1PRF1PRF1Open\-Source Models \(8B \- 671B\)Qwen3\-8B50\.1487\.2563\.6936\.8188\.1251\.9379\.7691\.1985\.0960\.5089\.8272\.30Qwen3\-32B42\.7691\.1858\.2233\.6491\.8849\.2580\.1294\.2886\.6256\.5493\.2070\.39Qwen3\-235B\-A22B\-Instruct40\.2596\.5556\.8128\.6595\.6244\.0966\.5599\.4879\.7548\.9998\.1965\.37Qwen3\-235B\-A22B\-Thinking37\.9497\.0154\.5528\.9595\.0044\.3871\.8498\.4483\.0750\.0397\.5566\.14DeepSeek\-V3\.232\.0596\.9748\.1826\.8396\.8642\.0265\.7599\.8379\.2844\.9798\.7161\.79Proprietary ModelsGPT\-4o50\.2986\.4363\.5932\.4886\.3947\.2169\.6096\.9881\.0455\.6792\.9569\.64doubao\-seed\-1\-6\-thinking41\.8195\.1058\.0837\.2996\.2553\.7574\.4798\.2684\.7355\.8397\.2370\.94claude\-4\-sonnet46\.3286\.2760\.2736\.6587\.5051\.6672\.9594\.9182\.4954\.8391\.2168\.49Open\-Source Models \(Qwen3\-4B based\)Qwen3\-4B60\.9271\.4365\.7643\.6673\.1254\.6784\.7586\.0985\.4160\.9280\.6774\.58HA\-GRM \(SFT\)78\.9156\.8666\.1071\.6169\.3870\.4881\.6152\.3365\.0981\.0456\.2066\.37HA\-GRM \(Response Only\)77\.4462\.2569\.0263\.7969\.3866\.4791\.2277\.3483\.7182\.7372\.7277\.40HA\-GRM \(Full\)81\.0560\.7869\.4770\.7565\.0067\.7591\.9877\.2083\.9485\.8871\.5878\.08Table 1:Performance \(Precision, Recall and F1\) of hallucination evaluation on the test split of RAGTruth\. We separate Qwen3\-4B based models, larger open\-source models, and proprietary models into different blocks\.We address the following research question: \(RQ1\) What are the advantages of using reinforcement learning with fine\-grained rewards for training generative reward models?

### 3\.1Experimental Setup

#### Training Details\.

We initialize HA\-GRM from Qwen3\-4B and train it on two public datasets\. Specifically, for hallucination detection, we use the summarization and question\-answering subsets of theRAGTruth[Niu et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib13)training set\. For creative writing, we utilizeArena\-Hard\-v2\.0\([Li et al\., 2025](https://arxiv.org/html/2610.03063#bib.bib11)\)\. We employ verl[Sheng et al\. \(2025\)](https://arxiv.org/html/2610.03063#bib.bib35)as our RL training framework, using a global batch size of 128 and a learning rate of 1e\-6\. Additionally, we apply a clipping strategy with the upper clipping threshold set to 0\.28\.

#### Evaluation Datasets and Metrics\.

We evaluate the reward models on the RAGTruth test set for hallucination detection and the Auto\-J pairwise preference evaluation data[Li et al\. \(2024a\)](https://arxiv.org/html/2610.03063#bib.bib36)for general preference evaluation\. In addition to response\-level metrics such as precision, recall, and F1 scores, we conduct span\-level detection by calculating the overlap between detected and human\-labeled spans, reporting the character\-level F1 score\. For the pairwise evaluation on Auto\-J, we report accuracy\. All datasets used in this evaluation are human\-annotated\.

### 3\.2Results and Analysis

#### Main Results\.

Table[1](https://arxiv.org/html/2610.03063#S3.T1)reports response\-level detection performance\. HA\-GRM \(Full\), initialized from Qwen3\-4B, achieves an Overall F1 of 78\.08%, compared with 74\.58% for the base model and 66\.37% for SFT\. The gains vary by task: Full has a higher summarization F1 than SFT \(69\.47% vs\. 66\.10%\), while SFT has a higher QA F1 \(70\.48% vs\. 67\.75%\)\. On D2T, which is excluded from our training data, Full reaches 83\.94%, above SFT’s 65\.09% but below the base model’s 85\.41%\. Full also has the highest reported Overall F1 among the compared models, while Qwen3\-32B retains the highest D2T F1 \(86\.62%\)\. These results support improved overall detection performance with task\-specific trade\-offs\.

#### Can span\-level hallucination prediction help?

Model NameSummQAD2TOverallQwen3\-4B41\.4334\.0038\.0837\.33HA\-GRM \(SFT\)45\.9461\.2630\.1344\.58HA\-GRM \(Response Only\)48\.5348\.7439\.3144\.43HA\-GRM \(Full\)52\.1053\.6347\.2650\.38

Table 2:Span\-level hallucination detection performance \(span\-F1\) on the RAGTruth test set\. Bold indicates the best score in each column\.To investigate span supervision, we compare HA\-GRM\(Full\)with\(Response Only\), which is trained solely with response\-level binary rewards\. In Table[2](https://arxiv.org/html/2610.03063#S3.T2), Full has higher span\-F1 on each subset than Response Only, with Overall span\-F1 increasing from 44\.43% to 50\.38%\. However, SFT retains the highest QA span\-F1 \(61\.26%, compared with Full’s 53\.63%\)\. Table[1](https://arxiv.org/html/2610.03063#S3.T1)also shows a response\-level Overall F1 increase from 77\.40% to 78\.08% for Full over Response Only\. Thus, span supervision improves the response\-only variant on the reported metrics, while SFT remains stronger on QA span detection\.

#### How does writing preference evaluation affect hallucination detection?

Model NameRAGTruthAuto\-JAverageQwen3\-4B74\.5857\.2265\.90w/ Hallucination78\.4157\.3167\.86w/ Hallucination and Writing78\.0859\.9169\.00

Table 3:Impact of multi\-objective training on hallucination detection \(RAGTruth\) and general preference evaluation \(Auto\-J\) test sets\.Table[3](https://arxiv.org/html/2610.03063#S3.T3)shows the trade\-off from adding writing preference data\. Compared with hallucination\-only training, joint training increases Auto\-J accuracy from 57\.31% to 59\.91%, while RAGTruth response\-level F1 decreases from 78\.41% to 78\.08%\. The reported cross\-task average rises from 67\.86% to 69\.00%\. Joint training therefore improves preference evaluation with a small reduction in hallucination detection F1; the two task metrics should be considered separately when choosing a training configuration\.

Takeaway \(RQ1\)HA\-GRM improves Overall response\-level F1 over the base model and SFT, with task\-specific trade\-offs\. Adding writing preference data raises Auto\-J accuracy with a small decrease in RAGTruth F1\.

## 4Experiments on HARPO

We aim to answer the research question: \(RQ2\) Can we leverage the feedback from HA\-GRM to effectively mitigate LLM hallucinations while incentivizing its creative writing capabilities?

### 4\.1Experimental Setup

#### Datasets and Evaluation Metrics\.

Our training corpus consists of three distinct datasets selected to balance faithful grounding with creative expression\. For hallucination mitigation, we utilize the summarization subset ofRAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.03063#bib.bib13)\)and the question\-answering subset ofHaluEval\([Li et al\., 2023](https://arxiv.org/html/2610.03063#bib.bib12)\)\. To support creative generation, we curate a subset of theArena\-Human\-Preference\-140kdataset from Chatbot Arena\([Chiang et al\., 2024](https://arxiv.org/html/2610.03063#bib.bib14)\),111[https://huggingface\.co/datasets/lmarena\-ai/arena\-human\-preference\-140k](https://huggingface.co/datasets/lmarena-ai/arena-human-preference-140k)filtering specifically for prompts related to creative writing\. All experiments are run once\.

For summarization, we evaluate on 831 source documents from the Vectara Hallucination Leaderboard dataset\([Hughes et al\., 2023](https://arxiv.org/html/2610.03063#bib.bib37)\), denotedHHEMin our tables\. For QA, we useMultiHopRAG\([Tang and Yang, 2024](https://arxiv.org/html/2610.03063#bib.bib38)\)\. Both hallucination rates are the percentage of generated responses judged hallucinated by our best\-performing HA\-GRM\. General and creative writing capabilities are evaluated on the Hard Prompts and Creative Writing subsets ofArena\-Hard\-v2\.0, respectively, using its official pipeline\([Li et al\., 2025](https://arxiv.org/html/2610.03063#bib.bib11)\)\.222[https://github\.com/lmarena/arena\-hard\-auto](https://github.com/lmarena/arena-hard-auto)Table[4](https://arxiv.org/html/2610.03063#S4.T4)summarizes the statistics for all datasets\.

DatasetDomain\# of SamplesTraining dataRAGTruthSummarization2,217HaluEvalQuestion Answering1,452Arena\-Human\-Preference\-140kCreative Writing3,000Testing dataHHEMSummarization831MultiHopRAGQuestion Answering2,556Arena\-Hard\-v2\.0 \(Hard Prompts\)General500Arena\-Hard\-v2\.0 \(Creative Writing\)Creative Writing250

Table 4:Statistics of the training and testing datasets utilized in the HARPO experiments\.
#### Baselines and Training Details\.

To evaluate HARPO, we compare it against three baselines: \(1\)Hallucination\-Only, trained exclusively onRAGTruthandHaluEvalusing onlyRhalluR\_\{\\text\{hallu\}\}; \(2\)Writing\-Only, trained solely onArena\-Human\-Preference\-140kusing onlyRcreatR\_\{\\text\{creat\}\}; and \(3\)Linear Mixture, a combined\-objective baseline\. Following[Peng et al\. \(2025\)](https://arxiv.org/html/2610.03063#bib.bib10), the Linear Mixture baseline performs multi\-objective alignment via fixed scalarization\. We normalize the writing reward to\[0,1\]\[0,1\]and compute the arithmetic mean with the hallucination reward:\(Rcreatnorm\+Rhallu\)/2\(R^\{\\text\{norm\}\}\_\{\\text\{creat\}\}\+R\_\{\\text\{hallu\}\}\)/2\. This serves as the primary state\-of\-the\-art comparison for mixed\-objective training\. This baseline uses the same training data as HARPO but a different reward strategy\. For reinforcement learning hyperparameters, we adopt the same configuration used to train HA\-GRM\.

Model NameHallucination Control↓\\downarrowCreative Writing↑\\uparrowGeneral↑\\uparrowHHEMMultiHopRAGArena\-Hard\-v2\.0\(Creative Writing\)Arena\-Hard\-v2\.0\(Hard Prompts\)Qwen2\.5 seriesQwen2\.5\-3B11\.074\.692\.332\.07w/ HARPO \(Ours\)2\.412\.862\.662\.30Qwen2\.5\-7B5\.051\.884\.684\.63w/ HARPO \(Ours\)1\.560\.866\.784\.93Qwen3 seriesQwen3\-1\.7B11\.075\.387\.043\.87w/ HARPO \(Ours\)3\.972\.038\.534\.05Qwen3\-4B7\.343\.2916\.9511\.16w/ Hallucination\-Only0\.720\.989\.745\.29w/ Writing\-Only24\.312\.6643\.3511\.45w/ Linear Mixture3\.730\.9028\.835\.63w/ HARPO \(Ours\)1\.081\.0227\.5412\.35Qwen3\-8B4\.932\.9033\.7616\.08w/ HARPO \(Ours\)1\.561\.1741\.4316\.52Table 5:Comparison of HARPO against base models and baselines\. \(↓\\downarrow\) indicates lower is better, \(↑\\uparrow\) indicates higher is better\. HARPO lowers HA\-GRM\-judged hallucination rates and improves writing scores relative to the corresponding base models\.

### 4\.2Results and Analysis

#### Main Results\.

Table[5](https://arxiv.org/html/2610.03063#S4.T5)presents the performance of HARPO across different model families \(Qwen2\.5, Qwen3\) and parameter scales \(1\.7B to 8B\)\. Our analysis yields two key observations:

1\. Trade\-offs in Single\-Objective Training:On Qwen3\-4B,Hallucination\-Onlyreaches the lowest HHEM hallucination rate \(0\.72%\), while Creative Writing decreases from 16\.95% to 9\.74% and General decreases from 11\.16% to 5\.29% relative to the base model\.Writing\-Onlyraises Creative Writing from 16\.95% to 43\.35%, while the HHEM hallucination rate rises from 7\.34% to 24\.31%\. These results show that optimizing either objective alone can improve its target metric while degrading another\.

2\. Improvements over Base Models:HARPO has lower HA\-GRM\-judged hallucination rates on both datasets and higher Creative Writing and General scores than the corresponding base models at every tested size\. For example, on Qwen2\.5\-7B, the HHEM hallucination rate decreases from 5\.05% to 1\.56% and Creative Writing increases from 4\.68% to 6\.78%\. These comparisons concern the base models; comparisons with other training strategies reveal additional trade\-offs\.

#### Comparison with Linear Reward Mixing\.

For Qwen3\-4B, HARPO has a lower HHEM hallucination rate thanLinear Mixture\(1\.08% vs\. 3\.73%\) and a higher General score \(12\.35% vs\. 5\.63%\)\. However, its MultiHopRAG hallucination rate is slightly higher \(1\.02% vs\. 0\.90%\), and its Creative Writing score is slightly lower \(27\.54% vs\. 28\.83%\)\. HARPO therefore improves summarization faithfulness as judged by HA\-GRM and performance on Hard Prompts, with a trade\-off in QA hallucination rate and creative\-writing score\. SAM activates writing rewards only for responses judged hallucination\-free by HA\-GRM; this comparison reflects the complete HARPO training configuration\.

#### Effect of Data Curriculum Strategy\.

Curriculum ScheduleHallucination Control↓\\downarrowCreative Writing↑\\uparrowHHEMMultiHopRAGArena\-Hard\-v2\.0\(Creative Writing\)Qwen3\-4B7\.343\.2916\.95Random Mixture \(Static\)4\.091\.1723\.65Faithful→\\toCreative \(Reverse\)9\.641\.7234\.30Creative→\\toFaithful \(Ours\)1\.081\.0227\.54Table 6:Ablation on Data Curriculum Strategies\.To validate our data curriculum strategy, we compare it against two alternative schedules as shown on the left of Figure[1](https://arxiv.org/html/2610.03063#S2.F1): \(1\)Random Mixture, which samples tasks with a fixed probability throughout training; and \(2\)Faithful→\\toCreative, a reverse schedule that begins with hallucination mitigation and transitions to creative writing\.

Table[6](https://arxiv.org/html/2610.03063#S4.T6)compares the reported curriculum outcomes\. Relative toRandom Mixture,Creative→\\toFaithfullowers the HHEM hallucination rate from 4\.09% to 1\.08% and the MultiHopRAG rate from 1\.17% to 1\.02%, while increasing Creative Writing from 23\.65% to 27\.54%\.

The reverseFaithful→\\toCreativeschedule obtains the highest Creative Writing score \(34\.30%\), but has higher hallucination rates on HHEM \(9\.64%\) and MultiHopRAG \(1\.72%\) thanCreative→\\toFaithful\(1\.08% and 1\.02%\)\. Thus, the proposed schedule favors lower HA\-GRM\-judged hallucination rates, with a lower creative\-writing score than the reverse schedule\. The preferred schedule depends on the relative priority of these metrics\.

![Refer to caption](https://arxiv.org/html/2610.03063v1/images/response_length_analysis_hhem.png)Figure 2:Response length analysis on HHEM\.
#### Analysis of Response Length and Verbosity

To investigate whether our framework mitigates “reward hacking”, specifically the tendency of RL\-tuned models to inflate scores through verbosity, we track the evolution of response length throughout the training process \(Figure[2](https://arxiv.org/html/2610.03063#S4.F2)\)\. TheWriting\-Onlybaseline exhibits a linear, unbounded increase in token count, confirming that optimizing solely for creativity encourages excessive length\. Conversely, theHallucination\-Onlybaseline rapidly converges to shorter, risk\-averse responses, potentially sacrificing necessary detail to minimize hallucination rate\. HARPO, however, demonstrates a distinct trajectory that balances these extremes\. Initially, the response length increases, aligning with the high ratio of creative tasks in the early curriculum\. However, as the training progresses beyond step 60, the length peaks and gradually stabilizes\. This shift corresponds directly with our curriculum schedule and the imposition of the Selective Activation Mechanism; as the distribution shifts toward faithful tasks and the SAM strictness takes effect, the model is regularized against gratuitous verbosity while retaining sufficient length to remain expressive\.

Takeaway \(RQ2\)HARPO improves the reported hallucination and writing metrics relative to the base models\. Compared with Linear Mixture on Qwen3\-4B, it has a lower HHEM hallucination rate and higher General score, with a slightly higher MultiHopRAG hallucination rate and lower Creative Writing score\.

## 5Related Work

#### Closed\-Domain Hallucination Detection

Closed\-domain hallucination refers to generating content that is inconsistent with or unsupported by the context or knowledge provided in the input prompt[Ji et al\. \(2023\)](https://arxiv.org/html/2610.03063#bib.bib1);[Niu et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib13)\. Traditional detection methods frame this as a Natural Language Inference \(NLI\) task[Lattimer et al\. \(2023\)](https://arxiv.org/html/2610.03063#bib.bib17);[Li et al\. \(2024b\)](https://arxiv.org/html/2610.03063#bib.bib18), where the source document serves as the premise and the generated text as the hypothesis\. As generative models improve, question answering provides another approach to hallucination detection[Honovich et al\. \(2021\)](https://arxiv.org/html/2610.03063#bib.bib19);[Cattan et al\. \(2026\)](https://arxiv.org/html/2610.03063#bib.bib20)\. More recent methods explore intrinsic LLM features, including internal activations and hidden\-state dynamics[Ridder and Schilling \(2024\)](https://arxiv.org/html/2610.03063#bib.bib22);[Zhang et al\. \(2025\)](https://arxiv.org/html/2610.03063#bib.bib21), as well as embedding distances[Ricco et al\. \(2025\)](https://arxiv.org/html/2610.03063#bib.bib23)\. Complementary resources include HalluLens for benchmarking hallucinations[Bang et al\. \(2025\)](https://arxiv.org/html/2610.03063#bib.bib15)and RAGTruth for training and evaluating detectors with response\- and span\-level annotations[Niu et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib13)\. RL4HS\([Su et al\., 2026](https://arxiv.org/html/2610.03063#bib.bib6)\)trains a reasoning\-based hallucination span detector using GRPO with verifiable span\-F1 rewards and introduces class\-aware policy optimization to address reward imbalance\. HARPO combines response\- and span\-level hallucination evaluation with writing preference evaluation in a generative reward model, then uses its feedback to optimize a generation policy through SAM and a creative\-to\-faithful data curriculum\.

#### Reinforcement Learning with Verifiable Rewards

Reinforcement Learning with Verifiable Rewards \(RLVR\) is a prominent method for enhancing LLM reasoning[Guo et al\. \(2025\)](https://arxiv.org/html/2610.03063#bib.bib2), initially successful in structured domains like mathematics and programming[Wen et al\. \(2026\)](https://arxiv.org/html/2610.03063#bib.bib3)using algorithms such as PPO[Schulman et al\. \(2017\)](https://arxiv.org/html/2610.03063#bib.bib4)and GRPO[Shao et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib5)\. RL4HS\([Su et al\., 2026](https://arxiv.org/html/2610.03063#bib.bib6)\)extends this approach to hallucination span detection using annotated spans as verifiable supervision\. In HARPO, we use verifiable hallucination labels, span annotations, and pairwise preference labels to train HA\-GRM; the resulting learned evaluator supplies rewards for downstream policy optimization\.

#### Joint Optimization of Writing Quality and Constraints

RLMR\([Liao et al\., 2026](https://arxiv.org/html/2610.03063#bib.bib7)\)combines subjective writing rewards with objective constraint verification and dynamically adjusts reward weights within sampled groups to penalize constraint violations\. HARPO uses predicted context faithfulness to gate the writing reward for each response, together with a curriculum that shifts training from creative to faithful tasks\. Our contribution centers on combining hallucination\-aware reward modeling with this conditional reward and data schedule to improve the balance between grounded generation and writing quality\.

#### Reducing Hallucination for LLMs

Research into mitigating LLM hallucinations spans data preparation, training, and inference\. RefinedWeb studies large\-scale filtering of web pretraining data[Penedo et al\. \(2023\)](https://arxiv.org/html/2610.03063#bib.bib24), while LIMA studies alignment through supervised fine\-tuning on a small curated dataset[Zhou et al\. \(2023\)](https://arxiv.org/html/2610.03063#bib.bib25)\. MOSS uses synthetic conversational data for instruction tuning[Sun et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib26), and Knowledge Consistent Alignment addresses inconsistencies between alignment data and a model’s knowledge[Wan et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib27)\. RLHF aligns models with human preferences[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.03063#bib.bib28), while process supervision evaluates intermediate reasoning steps[Lightman et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib16)\. TruthRL examines the trade\-off between correct answers, hallucinations, and abstention[Wei et al\. \(2026\)](https://arxiv.org/html/2610.03063#bib.bib34)\. Factuality Enhanced Language Models use factuality\-oriented continued training[Lee et al\. \(2022\)](https://arxiv.org/html/2610.03063#bib.bib29)\. At inference time, layer\-contrastive decoding[Chuang et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib30), context\-aware decoding[Shi et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib31), and Chain\-of\-Verification[Dhuliawala et al\. \(2024\)](https://arxiv.org/html/2610.03063#bib.bib32)reduce hallucinations through changes to decoding or response verification\.

## 6Conclusion

We presented HARPO, a reinforcement learning framework for joint optimization of hallucination mitigation and writing quality\. Its Selective Activation Mechanism activates writing rewards only for responses judged hallucination\-free by HA\-GRM, alongside a curriculum that shifts from creative to faithful tasks\. The reported results show lower HA\-GRM\-judged hallucination rates and higher writing scores relative to the base models\. Compared with Linear Mixture on Qwen3\-4B, HARPO lowers the summarization hallucination rate and raises the General score, with a slightly higher MultiHopRAG hallucination rate and lower Creative Writing score\. The curriculum results likewise show lower hallucination rates than the reverse schedule with a lower creative\-writing score\. These findings characterize the observed trade\-offs across training strategies\.

## Limitations

Despite the promising results of HARPO, several limitations remain\. First, training the Hallucination\-Aware Generative Reward Model \(HA\-GRM\) relies on high\-quality, span\-level hallucination annotations, which are significantly more labor\-intensive and costly to acquire than standard binary labels\. Second, the inference overhead introduced by the generative reward process increases the computational cost of training compared to traditional discriminator\-based reward models\. Finally, while we demonstrate effectiveness on models up to 8B parameters within general instruction and RAG domains, we have not yet verified the scalability of our framework on significantly larger models \(e\.g\., 70B\+\) or its transferability to specialized reasoning tasks such as mathematics and code generation, which we leave for future investigation\.

SAM depends on HA\-GRM’s predictions and provides no guarantee of factual correctness\. If HA\-GRM misses a hallucination, the affected response can still receive a writing reward; if it incorrectly flags a faithful response, that reward is suppressed\. These errors can affect policy optimization\. Since HA\-GRM also judges the reported generation hallucination rates, improvements on that metric should be interpreted with this shared evaluator in mind\.

## Ethics Statement

This study uses existing public datasets for model training and evaluation, including RAGTruth, HaluEval, Arena\-Human\-Preference\-140k, and Arena\-Hard\-v2\.0\. The RAGTruth and HaluEval repositories provide MIT licenses\.333[RAGTruth license](https://github.com/ParticleMedia/RAGTruth/blob/main/LICENSE);[HaluEval license](https://github.com/RUCAIBox/HaluEval/blob/main/LICENSE)\.Arena\-Human\-Preference\-140k licenses user prompts under CC BY 4\.0, while model outputs remain subject to the respective providers’ terms\.444[Arena\-Human\-Preference\-140k dataset card](https://huggingface.co/datasets/lmarena-ai/arena-human-preference-140k)\.The Arena\-Hard\-Auto dataset card lists Apache 2\.0\.555[Arena\-Hard\-Auto dataset card](https://huggingface.co/datasets/lmarena-ai/arena-hard-auto)\.Licenses and usage terms for underlying source corpora and pretrained models apply separately\.

Public user conversations and preference annotations may contain sensitive information and reflect contributor biases\. Reuse and redistribution should avoid disclosing personal information and must retain attribution and applicable usage restrictions\.

HA\-GRM can miss unsupported claims or incorrectly flag valid content, and these errors can affect both policy optimization and reported hallucination rates\. Preference optimization may also favor writing styles represented in the training data\. The experiments do not establish safety or reliability for high\-stakes applications; deployment requires independent evaluation and appropriate human oversight\.

## References

- Banget al\.\(2025\)Y\. Bang, Z\. Ji, A\. Schelten, A\. Hartshorn, T\. Fowler, C\. Zhang, N\. Cancedda, and P\. FungHalluLens: LLM Hallucination Benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 24128–24156\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1176),[Link](https://aclanthology.org/2025.acl-long.1176/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Cattanet al\.\(2026\)A\. Cattan, P\. Roit, S\. Zhang, D\. Wan, R\. Aharoni, I\. Szpektor, M\. Bansal, and I\. DaganLocalizing Factual Inconsistencies in Attributable Text Generation\.Transactions of the Association for Computational Linguistics14,pp\. 100–123\.External Links:[Document](https://dx.doi.org/10.1162/tacl.a.598),[Link](https://aclanthology.org/2026.tacl-1.6/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Chianget al\.\(2024\)W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, B\. Zhu, H\. Zhang, M\. Jordan, J\. E\. Gonzalez, and I\. StoicaChatbot Arena: An Open Platform for Evaluating LLMs by Human Preference\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 8359–8388\.External Links:[Link](https://proceedings.mlr.press/v235/chiang24b.html)Cited by:[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px1.p1.1)\.
- Chuanget al\.\(2024\)Y\. Chuang, Y\. Xie, H\. Luo, Y\. Kim, J\. Glass, and P\. HeDoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/edc36117f795ca52a0cbf6a7b3882859-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Damaniet al\.\(2026\)M\. Damani, I\. Puri, S\. Slocum, I\. Shenfeld, L\. Choshen, Y\. Kim, and J\. AndreasBeyond Binary Rewards: Training LMs to Reason About Their Uncertainty\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/615675cc6e94ddb1a783904fb178b5f6-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.03063#S1.p2.1)\.
- Dhuliawalaet al\.\(2024\)S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. WestonChain\-of\-Verification Reduces Hallucination in Large Language Models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 3563–3578\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.212),[Link](https://aclanthology.org/2024.findings-acl.212/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645,pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),[Link](https://www.nature.com/articles/s41586-025-09422-z)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px2.p1.1)\.
- Honovichet al\.\(2021\)O\. Honovich, L\. Choshen, R\. Aharoni, E\. Neeman, I\. Szpektor, and O\. AbendQ2\{\}^\{2\}: Evaluating Factual Consistency in Knowledge\-Grounded Dialogues via Question Generation and Question Answering\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 7856–7870\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.619),[Link](https://aclanthology.org/2021.emnlp-main.619/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Hugheset al\.\(2023\)S\. Hughes, M\. Bae, and M\. LiVectara Hallucination Leaderboard\.Note:DatasetExternal Links:[Link](https://github.com/vectara/hallucination-leaderboard)Cited by:[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px1.p2.1)\.
- Jiet al\.\(2023\)Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. FungSurvey of Hallucination in Natural Language Generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730),[Link](https://doi.org/10.1145/3571730)Cited by:[§1](https://arxiv.org/html/2610.03063#S1.p1.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Lattimeret al\.\(2023\)B\. Lattimer, P\. H\. Chen, X\. Zhang, and Y\. YangFast and Accurate Factual Inconsistency Detection Over Long Documents\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 1691–1703\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.105),[Link](https://aclanthology.org/2023.emnlp-main.105/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Leeet al\.\(2022\)N\. Lee, W\. Ping, P\. Xu, M\. Patwary, P\. N\. Fung, M\. Shoeybi, and B\. CatanzaroFactuality Enhanced Language Models for Open\-Ended Text Generation\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 34586–34599\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/df438caa36714f69277daa92d608dd63-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2024a\)J\. Li, S\. Sun, W\. Yuan, R\. Fan, H\. Zhao, and P\. LiuGenerative Judge for Evaluating Alignment\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/747dc7c6566c74eb9a663bcd8d057c78-Abstract-Conference.html)Cited by:[§3\.1](https://arxiv.org/html/2610.03063#S3.SS1.SSS0.Px2.p1.1)\.
- Liet al\.\(2023\)J\. Li, X\. Cheng, X\. Zhao, J\. Nie, and J\. WenHaluEval: A Large\-Scale Hallucination Evaluation Benchmark for Large Language Models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 6449–6464\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.397),[Link](https://aclanthology.org/2023.emnlp-main.397/)Cited by:[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2024b\)M\. Li, R\. Luo, and O\. MendelevitchHHEM\-2\.1\-Open\.Hugging Face\.External Links:[Document](https://dx.doi.org/10.57967/hf/3240),[Link](https://huggingface.co/vectara/hallucination_evaluation_model)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. StoicaFrom Crowdsourced Data to High\-quality Benchmarks: Arena\-Hard and Benchbuilder Pipeline\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 34209–34231\.External Links:[Link](https://proceedings.mlr.press/v267/li25h.html)Cited by:[§2\.1](https://arxiv.org/html/2610.03063#S2.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.03063#S3.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px1.p2.1)\.
- Liaoet al\.\(2026\)J\. Liao, T\. Zhang, X\. Feng, Y\. Zhang, H\. Wang, B\. Wen, Z\. Wang, and R\. ShiRLMR: Reinforcement Learning with Mixed Rewards for Creative Writing\.Proceedings of the AAAI Conference on Artificial Intelligence40\(38\),pp\. 31970–31978\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i38.40467),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40467)Cited by:[§1](https://arxiv.org/html/2610.03063#S1.p2.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px3.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s Verify Step by Step\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/aca97732e30bcf1303bc22ac3924fd16-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Niuet al\.\(2024\)C\. Niu, Y\. Wu, J\. Zhu, S\. Xu, K\. Shum, R\. Zhong, J\. Song, and T\. ZhangRAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval\-Augmented Language Models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10862–10878\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.585),[Link](https://aclanthology.org/2024.acl-long.585/)Cited by:[§2\.1](https://arxiv.org/html/2610.03063#S2.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.03063#S3.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.External Links:[Link](https://proceedings.neurips.cc/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Penedoet al\.\(2023\)G\. Penedo, Q\. Malartic, D\. Hesslow, R\. Cojocaru, H\. Alobeidli, A\. Cappelli, B\. Pannier, E\. Almazrouei, and J\. LaunayThe RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 79155–79172\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2023/hash/fa3ed726cc5073b9c31e3e49a807789c-Abstract-Datasets_and_Benchmarks.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Penget al\.\(2025\)H\. Peng, Y\. Qi, X\. Wang, Z\. Yao, B\. Xu, L\. Hou, and J\. LiAgentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems\.arXiv preprint arXiv:2502\.19328\.External Links:[Link](https://arxiv.org/abs/2502.19328)Cited by:[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px2.p1.1)\.
- Riccoet al\.\(2025\)E\. Ricco, L\. Cima, and R\. Di PietroHallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis\.arXiv preprint arXiv:2502\.08663\.External Links:[Link](https://arxiv.org/abs/2502.08663)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Ridder and Schilling \(2024\)F\. Ridder and M\. SchillingThe HalluRAG Dataset: Detecting Closed\-Domain Hallucinations in RAG Applications Using an LLM’s Internal States\.arXiv preprint arXiv:2412\.17056\.External Links:[Link](https://arxiv.org/abs/2412.17056)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal Policy Optimization Algorithms\.arXiv preprint arXiv:1707\.06347\.External Links:[Link](https://arxiv.org/abs/1707.06347)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2610.03063#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.03063#S2.SS1.SSS0.Px2.p1.1),[§2\.2](https://arxiv.org/html/2610.03063#S2.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px2.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: A Flexible and Efficient RLHF Framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.External Links:[Document](https://dx.doi.org/10.1145/3689031.3696075),[Link](https://doi.org/10.1145/3689031.3696075)Cited by:[§3\.1](https://arxiv.org/html/2610.03063#S3.SS1.SSS0.Px1.p1.1)\.
- Shiet al\.\(2024\)W\. Shi, X\. Han, M\. Lewis, Y\. Tsvetkov, L\. Zettlemoyer, and W\. YihTrusting Your Evidence: Hallucinate Less with Context\-aware Decoding\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 783–791\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-short.69),[Link](https://aclanthology.org/2024.naacl-short.69/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Suet al\.\(2026\)H\. Su, T\. Hu, H\. S\. Koppula, K\. Krishna, H\. Pouransari, C\. Hsieh, C\. Koc, J\. Y\. Cheng, O\. Tuzel, and R\. VemulapalliLearning to Reason for Hallucination Span Detection\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/5b0c1a9ce8c71654833210865a256161-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.03063#S1.p4.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px2.p1.1)\.
- Sunet al\.\(2024\)T\. Sun, X\. Zhang, Z\. He, P\. Li, Q\. Cheng, X\. Liu, H\. Yan, Y\. Shao, Q\. Tang, S\. Zhang,et al\.MOSS: An Open Conversational Large Language Model\.Machine Intelligence Research21\(5\),pp\. 888–905\.External Links:[Document](https://dx.doi.org/10.1007/s11633-024-1502-8),[Link](https://doi.org/10.1007/s11633-024-1502-8)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Tang and Yang \(2024\)Y\. Tang and Y\. YangMultiHop\-RAG: Benchmarking Retrieval\-Augmented Generation for Multi\-Hop Queries\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=t4eB3zYWBK)Cited by:[§4\.1](https://arxiv.org/html/2610.03063#S4.SS1.SSS0.Px1.p2.1)\.
- Wanet al\.\(2024\)F\. Wan, X\. Huang, L\. Cui, X\. Quan, W\. Bi, and S\. ShiKnowledge Verification to Nip Hallucination in the Bud\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2616–2633\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.152),[Link](https://aclanthology.org/2024.emnlp-main.152/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Weiet al\.\(2026\)Z\. Wei, X\. Yang, K\. Sun, J\. Wang, R\. Shao, J\. Chen, M\. Kachuee, T\. Gollapudi, Y\. Liao, N\. Scheffer,et al\.TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 134156–134175\.External Links:[Link](https://proceedings.mlr.press/v306/wei26ab.html)Cited by:[§1](https://arxiv.org/html/2610.03063#S1.p2.1),[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.
- Wenet al\.\(2026\)X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao,et al\.Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/517f9b9c227b9dd51dba4560f37165ed-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px2.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3\-4\),pp\. 229–256\.External Links:[Document](https://dx.doi.org/10.1007/bf00992696),[Link](https://doi.org/10.1007/bf00992696)Cited by:[§2\.2](https://arxiv.org/html/2610.03063#S2.SS2.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu,et al\.DAPO: An Open\-Source LLM Reinforcement Learning System at Scale\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 113222–113244\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2610.03063#S2.SS2.SSS0.Px1.p3.3)\.
- Zhanget al\.\(2025\)Z\. Zhang, X\. Hu, H\. Zhang, J\. Zhang, and X\. WanICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17986–18002\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.880),[Link](https://aclanthology.org/2025.acl-long.880/)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2023\)C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu,et al\.LIMA: Less Is More for Alignment\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 55006–55021\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/ac662d74829e4407ce1d126477f4a03a-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.03063#S5.SS0.SSS0.Px4.p1.1)\.