Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

arXiv cs.CL Papers
hallucination reasoning-models reinforcement-learning factuality large-language-models

Summary

Introduces MARGO, a reinforcement learning framework that uses mixed-mode advantage regularization to mitigate thinking-induced hallucinations in large reasoning models by comparing thinking and non-thinking trajectories.

arXiv:2607.05861v1 Announce Type: new Abstract: Large reasoning models (LRMs) improve language model capabilities by generating explicit thinking traces before final answers. In factuality-oriented question answering (QA), such thinking often improves overall performance by helping the model recover relevant knowledge and refine its answers. However, we find that this benefit is not uniform at the instance level: explicit thinking can also overturn correct non-thinking answers and lead to factual drift. We refer to this failure mode as \emph{thinking-induced hallucination}. To explain this phenomenon, we formulate explicit thinking in factuality QA as a thinking residual over the model's direct-answer tendency, which can either recover missing knowledge or introduce unsupported associations. Based on this formulation, we propose MARGO, \underline{\textit{M}}ixed-Mode \underline{\textit{A}}dvantage \underline{\textit{R}}egularization for \underline{\textit{G}}rounded \underline{\textit{O}}ptimization, a reinforcement learning framework that uses non-thinking rollouts as same-model references in advantage estimation. By constructing mixed-mode rollout groups with both thinking and non-thinking trajectories, MARGO evaluates whether explicit thinking adds factual value beyond direct answering, thereby suppressing hallucination-prone thinking while preserving beneficial thinking behaviors. Experiments across multiple factuality-oriented QA benchmarks demonstrate that MARGO improves factual reliability over strong baselines, while evaluations on mathematical benchmarks show that it preserves general reasoning ability.
Original Article
View Cached Full Text

Cached at: 07/08/26, 04:41 AM 

# Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization
Source: [https://arxiv.org/abs/2607.05861](https://arxiv.org/abs/2607.05861)
[View PDF](https://arxiv.org/pdf/2607.05861)

> Abstract:Large reasoning models \(LRMs\) improve language model capabilities by generating explicit thinking traces before final answers\. In factuality\-oriented question answering \(QA\), such thinking often improves overall performance by helping the model recover relevant knowledge and refine its answers\. However, we find that this benefit is not uniform at the instance level: explicit thinking can also overturn correct non\-thinking answers and lead to factual drift\. We refer to this failure mode as \\emph\{thinking\-induced hallucination\}\. To explain this phenomenon, we formulate explicit thinking in factuality QA as a thinking residual over the model's direct\-answer tendency, which can either recover missing knowledge or introduce unsupported associations\. Based on this formulation, we propose MARGO, \\underline\{\\textit\{M\}\}ixed\-Mode \\underline\{\\textit\{A\}\}dvantage \\underline\{\\textit\{R\}\}egularization for \\underline\{\\textit\{G\}\}rounded \\underline\{\\textit\{O\}\}ptimization, a reinforcement learning framework that uses non\-thinking rollouts as same\-model references in advantage estimation\. By constructing mixed\-mode rollout groups with both thinking and non\-thinking trajectories, MARGO evaluates whether explicit thinking adds factual value beyond direct answering, thereby suppressing hallucination\-prone thinking while preserving beneficial thinking behaviors\. Experiments across multiple factuality\-oriented QA benchmarks demonstrate that MARGO improves factual reliability over strong baselines, while evaluations on mathematical benchmarks show that it preserves general reasoning ability\.

## Submission history

From: Kaishen Wang \[[view email](https://arxiv.org/show-email/6224f4ff/2607.05861)\] **\[v1\]**Tue, 7 Jul 2026 05:34:25 UTC \(1,065 KB\)

Similar Articles

Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors

arXiv cs.CL

This paper studies why language models hallucinate, proposing that hallucinations often stem from biased latent inference (inference misalignment) rather than missing knowledge. It introduces TrapQA, a controlled diagnostic testbed to test reasoning against priors, and demonstrates that hallucinations can arise from misleading latent associations.

Learning to Reason for Factuality

arXiv cs.CL

This paper proposes a novel online reinforcement learning method to improve factuality in reasoning LLMs by designing a reward function that balances factual precision, detail, and relevance, achieving a 23.1 percentage point reduction in hallucination rate on six benchmarks.

RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration

arXiv cs.CL

RAGognizer introduces a hallucination-aware fine-tuning approach that integrates a lightweight detection head into LLMs for joint optimization of language modeling and hallucination detection in RAG systems. The paper presents RAGognize, a dataset of naturally occurring closed-domain hallucinations with token-level annotations, and demonstrates state-of-the-art hallucination detection while reducing hallucination rates without degrading language quality.