SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Hugging Face Daily Papers Papers
multi-task-learning sft reinforcement-learning llm gradient-interference theoretical-analysis

Summary

This paper presents a theoretical and empirical analysis showing that SFT suffers from task conflicts in multi-task LLM training while RL enables stable coexistence, proposing the Parallel-RL paradigm for efficient multi-task training.

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.
Original Article
View Cached Full Text

Cached at: 08/10/26, 10:14 AM

Paper page - SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Source: https://huggingface.co/papers/2608.03573

Abstract

SupervisedFine-Tuning(SFT)andReinforcementLearning(RL)exhibitfundamentallydifferentbehaviorsinenhancingmulti-taskreasoningforlargelanguagemodels(LLMs).Ourpreliminaryexperimentsrevealedaphenomenon:SFTsuffersfromseveretaskconflictsundermulti-stagetraining,whereasRLenablesstablecoexistenceacrossdiversetasks.Empirically,wetracethistotheparameterlevel,observingthatRLinducessparseandapproximatelyorthogonalupdatesacrosstasks.Weprovideatheoreticalexplanationforthismechanismbyanalyzingmulti-taskgradientinterference.Ourresultsrevealadistinction:interferenceinSFTisnorm-limited,scalingwiththeabsolutegradientmagnitude,whereasinterferenceinRLisvariance-limited,boundedbythegradientvarianceinducedbyadvantagenormalizationandon-policyoptimization.Thissmallvarianceboundyieldsnear-orthogonaloptimizationdirectionsacrosstasks.Leveragingthisinsight,weproposeParallel-RL,aparadigmthatdecouplesmulti-tasktraining,significantlyimprovingefficiencyandflexibility.

View arXiv pageView PDFGitHub5Add to collection

Get this paper in your agent:

hf papers read 2608\.03573

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.03573 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.03573 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.03573 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

arXiv cs.LG

Harvard researchers challenge the standard LLM training pipeline by showing RL can be effectively applied during pre-training rather than only after SFT, finding that data composition matters more than model scale, and proposing parallel averaging of RL and SFT objectives that outperforms sequential approaches while preserving general capabilities.

Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning

arXiv cs.CL

This paper proposes a difficulty-aware SFT-then-RL framework for training small language models (≤3B parameters) on reasoning tasks, arguing that data difficulty should be strategically aligned with the distinct roles of SFT (learning new skills) and RL (consolidating partial skills). The authors introduce a Bridge mechanism for hard SFT samples and Critique Fine-Tuning for RL failures, showing consistent improvements across five reasoning benchmarks.

@rohanpaul_ai: New Stanford paper argues that, under equal reasoning budgets, one LLM usually solves multi-hop problems better than ma…

X AI KOLs Timeline

A new Stanford paper shows that under equal reasoning token budgets, single LLMs typically outperform multi-agent systems on multi-hop reasoning tasks, with gains from multi-agent setups often stemming from additional compute rather than architectural superiority. The paper uses the Data Processing Inequality to explain why information loss in handoffs harms multi-agent performance, and identifies context quality as the key factor where multi-agent systems can provide benefits.