ISO: An RLVR-Native Optimization Stack

Hugging Face Daily Papers Papers
reinforcement-learning optimization rlvr spectral-inheritance isospectral language-models reasoning

Summary

This paper studies the optimization layer for reinforcement learning with verifiable rewards (RLVR), proposing Isospectral Optimization (ISO) — a fixed-spectrum framework that reuses base model weight spectra while optimizing input/output singular frames. ISO-Merger and ISO-AdamW achieve strong performance with fewer training steps on reasoning and coding tasks.

Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through the singular structure of model weights and identify spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior through changes in the associated input and output singular frames. We operationalize spectral inheritance as Isospectral Optimization (ISO), an RLVR-native, fixed-spectrum optimization framework with complementary offline and online instantiations. Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model, requiring no post-merge data, rollouts, gradient updates, or on-policy distillation (OPD). It recovers complementary specialist capabilities and achieves the strongest aggregate performance among the compared data-free merging methods. Online, ISO-Optimizer applies a chosen base optimizer, including AdamW and Muon, to the frame variables while keeping the base spectra fixed. Across reasoning and coding tasks ranging from 1.5B to 8B parameters, ISO-Optimizer improves accuracy in the reported runs and reaches matched scores with substantially fewer training steps. On Qwen3-8B-Base, AdamW reaches an aggregate accuracy of 0.495 after 270 training steps. ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 after 210 training steps. Together, ISO offers a concrete answer to RLVR's missing optimization layer: rather than inheriting pre-training optimization wholesale, design post-training around the structure of reward-driven adaptation: inherit the spectrum, optimize the frames.
Original Article
View Cached Full Text

Cached at: 07/22/26, 06:41 AM

Paper page - ISO: An RLVR-Native Optimization Stack

Source: https://huggingface.co/papers/2607.19331 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Reinforcementlearningwithverifiablerewards(RLVR)israpidlyadvancingthereasoningcapabilitiesoflanguagemodels,yettheoptimizationlayerthatconvertsrewardfeedbackintoweight-spaceupdatesremainspoorlyunderstood.Buildingonourprioranalysis(Zhuetal.,2025),westudythismissinglayerthroughthesingularstructureofmodelweightsandidentifyspectralinheritance:RLVRcanreusethebasemodel’sweightspectrawhileacquiringnewbehaviorthroughchangesintheassociatedinputandoutputsingularframes.WeoperationalizespectralinheritanceasIsospectralOptimization(ISO),anRLVR-native,fixed-spectrumoptimizationframeworkwithcomplementaryofflineandonlineinstantiations.Offline,ISO-Mergercombinestheframechangesofshared-basespecialistsintoasinglefixed-spectrummodel,requiringnopost-mergedata,rollouts,gradientupdates,oron-policydistillation(OPD).Itrecoverscomplementaryspecialistcapabilitiesandachievesthestrongestaggregateperformanceamongthecompareddata-freemergingmethods.Online,ISO-Optimizerappliesachosenbaseoptimizer,includingAdamWandMuon,totheframevariableswhilekeepingthebasespectrafixed.Acrossreasoningandcodingtasksrangingfrom1.5Bto8Bparameters,ISO-Optimizerimprovesaccuracyinthereportedrunsandreachesmatchedscoreswithsubstantiallyfewertrainingsteps.OnQwen3-8B-Base,AdamWreachesanaggregateaccuracyof0.495after270trainingsteps.ISO-AdamWreachesthesameaccuracyafteronly100trainingstepsandimprovesfurtherto0.509after210trainingsteps.Together,ISOoffersaconcreteanswertoRLVR’smissingoptimizationlayer:ratherthaninheritingpre-trainingoptimizationwholesale,designpost-trainingaroundthestructureofreward-drivenadaptation:inheritthespectrum,optimizetheframes.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.19331

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.19331 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.19331 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.19331 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization

arXiv cs.LG

This paper introduces Interactive Inverse Reinforcement Learning (IIRL), a framework where a learner actively interacts with an expert to infer reward functions, formulated as a stochastic bi-level optimization problem. The authors propose the BISIRL algorithm, providing convergence guarantees and experimental validation for this interactive learning paradigm.

Structured Role-Aware Policy Optimization for Multimodal Reasoning

arXiv cs.AI

This paper introduces Structured Role-Aware Policy Optimization (SRPO), a method that improves multimodal reasoning in Large Vision-Language Models by assigning token-level credit based on distinct perception and reasoning roles within reinforcement learning frameworks.