Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

Hugging Face Daily Papers Papers
tool-use reinforcement-learning large-language-models catastrophic-collapse supervisory-signals fine-tuning

Summary

This paper investigates why multi-step tool-use reinforcement learning leads to catastrophic collapse in LLMs and how interleaving supervised fine-tuning with diverse supervisory signals can stabilize training.

Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.
Original Article
View Cached Full Text

Cached at: 06/26/26, 06:05 AM

Paper page - Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

Source: https://huggingface.co/papers/2606.26027

Abstract

Research investigates how different supervisory signals and training strategies improve the stability and performance of large language models in tool-use tasks, addressing issues like catastrophic collapse and format sensitivity through interleaved supervised fine-tuning and reinforcement learning.

Tool use enables large language models (LLMs) to perform complex tasks, and recentagentic reinforcement learning(RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains intool-use tasks. In our experiments, some models exhibitcatastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specificcontrol tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, includingoff-policy supervision,hint-based guidance,erroneous example supervision, and others, applied under both synchronous andinterleaved trainingschemes. We find that interleavingsupervised fine-tuning(SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact oflearning ratesand generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guideexploratory learning, enabling robust training of LLMs for complex, multi-steptool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2606\.26027

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.26027 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.26027 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.26027 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles