Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models

Hugging Face Daily Papers Papers
reinforcement-learning retrieval-augmented-generation search-agents llm-reasoning multi-hop-reasoning deep-research

Summary

提出 Self-Assessment 强化学习框架（SARL），让检索增强的 LLM 搜索智能体在训练中学会感知自身搜索状态、识别缺失信息，从而提升搜索效率而无需额外推理时评估，在多跳推理与深度研究基准上取得显著相对提升。

Recent advances in search agents have enabled large language models to tackle complex tasks by retrieving and synthesizing external information. However, existing search agents often lack search-state awareness, i.e., the ability to recognize what information has been obtained and what remains missing, leading to hallucinated reasoning and prematurely terminated search. To address this limitation, we propose Self-Assessment Reinforcement Learning (SARL), a reinforcement learning framework in which models learn to recognize their own search state during training and improve their search behavior accordingly, without requiring additional assessment at test time. SARL achieves this through a dual reward design that captures the corrective effect of each assessment while penalizing excessive reliance on it, jointly encouraging effective and efficient search trajectories. Under this setup, SARL achieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% on multi-hop reasoning and 27.9% on deep research benchmarks against the strongest per-benchmark baselines. SARL further surpasses methods built on auxiliary supervision or multi-agent coordination, despite operating as a standalone RL framework with no inference-time scaffolding. Beyond final performance, SARL maintains training stability under extended search horizons, a regime where existing methods consistently degrade, enabling sustained gains as task complexity grows.
Original Article
View Cached Full Text

Cached at: 10/29/26, 01:32 AM

Paper page - Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models

Source: https://huggingface.co/papers/2610.30435

Abstract

Self-Assessment Reinforcement Learning trains search agents to recognize missing information and improve search efficiency without extra inference-time assessment.

Recent advances in search agents have enabled large language models to tackle complex tasks by retrieving and synthesizing external information. However, existing search agents often lacksearch-state awareness, i.e., the ability to recognize what information has been obtained and what remains missing, leading to hallucinated reasoning and prematurely terminated search. To address this limitation, we proposeSelf-Assessment Reinforcement Learning(SARL), areinforcement learningframework in which models learn to recognize their own search state during training and improve their search behavior accordingly, without requiring additional assessment at test time.SARLachieves this through adual reward designthat captures the corrective effect of each assessment while penalizing excessive reliance on it, jointly encouraging effective and efficient search trajectories. Under this setup,SARLachieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% onmulti-hop reasoningand 27.9% ondeep researchbenchmarks against the strongest per-benchmark baselines.SARLfurther surpasses methods built on auxiliary supervision or multi-agent coordination, despite operating as a standalone RL framework with no inference-time scaffolding. Beyond final performance,SARLmaintains training stability under extended search horizons, a regime where existing methods consistently degrade, enabling sustained gains as task complexity grows.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.30435

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### Taewhoo/self-assessment-qwen2.5-7b-step325 Text Generation• 8B• Updated1 day ago • 102

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.30435 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.30435 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Capability Self-Assessment: Teaching LLMs to Know Their Limits

arXiv cs.AI

This paper introduces Capability Self-Assessment (CSA) for LLMs, formulating it as a policy-learning problem. Experiments show that reinforcement learning effectively teaches models to recognize their own limits and delegate queries they cannot solve, outperforming supervised fine-tuning and generalizing well out-of-distribution.

@alacheng: The DeepSeek V4.1 flash 1017 update mentions a Reinforcement Learning (RL) training technique based on self-feedback. Reinforcement Learning with Self-Feedback means: during RL training, instead of relying entirely on external "ground truth" or human-annotated rewards…

X AI KOLs Timeline

The article unpacks the "Reinforcement Learning with Self-Feedback" technique mentioned in the DeepSeek V4.1 flash update, explains how it provides denser reward signals for long-horizon agent tasks that lack ground truth answers — such as "wish-based programming" — and compiles a list of 10 papers on self-feedback reinforcement learning from 2024–2026.