From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Hugging Face Daily Papers Papers
reinforcement-learning llm self-improvement verifiable-rewards task-transformation multi-agent self-play

Summary

This paper proposes RLSVR, a task-transformation paradigm that extends reinforcement learning with verifiable rewards to open-ended LLM tasks by creating self-verifiable proxy environments, instantiated via the SpyRL multi-agent self-play framework, showing gains on summarization, creative writing, and math reasoning.

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verified. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs.Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a multi-agent self-play environment inspired by Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/SpyRL.
Original Article
View Cached Full Text

Cached at: 08/03/26, 05:30 AM

Paper page - From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Source: https://huggingface.co/papers/2607.23802

Abstract

ReinforcementLearningwithVerifiableRewards(RLVR)hasdrivenrecentprogressinreasoning-orientedlargelanguagemodels(LLMs)byenablinglarge-scaleoptimization.However,itsapplicabilityremainslargelylimitedtodomainssuchasmathematicsandcoding,wherecorrectnesscanbedeterministicallyverified.Open-endedtasksinsteadoftenrelyonhumanpreferences,rewardmodels,orLLM-basedjudges,introducingevaluationbias,judgecapabilitybottlenecks,andadditionalinferencecosts.Drawingontheprincipleofself-supervisedlearning,whichconstructspretexttaskstoderivesupervisionfromthedataitself,weproposeReinforcementLearningwithSelf-VerifiableRewards(RLSVR),atask-transformation-basedtrainingparadigmforextendingRLVRtoopen-endedtasks.RLSVRtransformsopen-endedtasksintoverifiableproxyenvironmentswhoseinternalrulesandinteractionoutcomesautomaticallygeneraterewardsignals.WeinstantiateRLSVRwithSpyRL,amulti-agentself-playenvironmentinspiredbyWhoIstheSpy?.Agentsreceiveasymmetricinformation,completethesametargettask,andvotetoidentifyadesignatedspy.Becausethespyidentityispredetermined,votingoutcomesprovidefullyverifiablerewards,whilesuccessfulidentificationremainscloselyrelatedtooutputquality.Experimentsontextsummarization,creativewriting,andmathematicalreasoningshowthatSpyRLoutperformsexistingself-improvementmethodsonnon-verifiabletasksandyieldsconsistentgainsonverifiablereasoningtasks.TheseresultsdemonstratethattasktransformationcanextendscalableRLVR-basedself-improvementbeyondinherentlyverifiabledomains.Modelsandcodehavebeenreleasedathttps://github.com/wangqinsi1/SpyRL.

View arXiv pageView PDFProject pageGitHubAdd to collection

Get this paper in your agent:

hf papers read 2607\.23802

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.23802 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.23802 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.23802 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

From RLVR to RLSVR (GitHub Repo)

TLDR AI

Introduces RLSVR, a task-transformation paradigm that extends RLVR to open-ended tasks via self-verifiable rewards from self-play games, instantiated in SpyRL and Vision-Zero. It improves LLM performance on summarization, creative writing, and math reasoning.

RL Beyond the Verifiable (8 minute read)

TLDR AI

An analysis discussing the limitations of reinforcement learning with verifiable rewards (RLVR) in math and coding, and the challenge of extending RL to subjective or unverifiable tasks like planning or scientific discovery. It explores techniques such as RLHF and Constitutional AI as alternatives for alignment.

Video Models Can Reason with Verifiable Rewards

Hugging Face Daily Papers

VideoRLVR optimizes video diffusion models for verifiable reasoning tasks using reinforcement learning with rule-based rewards, achieving better performance than supervised methods in constraint-satisfying video generation.