SLPO: Scaling Latent Reasoning via a Surrogate Policy

Hugging Face Daily Papers Papers
latent-reasoning reinforcement-learning surrogate-policy test-time-scaling verifiable-rewards autoregressive-models

Summary

Introduces Surrogate Latent Policy Optimization (SLPO) to apply outcome-reward RL to autoregressive latent reasoners, enabling test-time scaling and variable-horizon policies that improve accuracy on harder instances.

Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@k under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.
Original Article
View Cached Full Text

Cached at: 07/24/26, 05:09 AM

Paper page - SLPO: Scaling Latent Reasoning via a Surrogate Policy

Source: https://huggingface.co/papers/2607.19691

Abstract

Reinforcementlearningwithverifiablerewardshasbecomethepredominantrecipeforelicitingtest-timescalinginexplicitChain-of-Thoughtreasoners.Yetthisscalingpathremainscomputationallycostly,sinceeveryintermediatestepmustbedecodedasalanguagetoken.LatentreasoninginsteadcarriesintermediatecomputationascontinuousvectorsandalreadymatchesorsurpassesexplicitCoTatfarshorterhorizons.Despitethispromise,latentreasonersremainlargelyimitation-bound,whileexplicitCoThasalreadymovedpastimitationviaoutcome-rewardRL.Latenttrajectorieslackatractableper-steplikelihoodandanadaptivestoppinginterfaceunderfixedthinkingbudgets,sooutcomerewardscannotelicitlatenttest-timescaling.WeintroduceSurrogateLatentPolicyOptimization(SLPO)tobringoutcome-rewardRLtoautoregressivelatentreasoners:anempiricalsurrogatepolicydensityoverlatenttransitionsfortrajectory-levelcreditassignment,andacorrectness-supervisedstoppingheadthatoutcome-rewardoptimizationrefinesintoavariable-horizonpolicy.Acrosscontinuousandsoftthinkingsettings,SLPOimprovesPass@kunderparallelsamplingandallocateslongerlatentcomputationtoharderinstanceswithhigherdeterministicaccuracy.

View arXiv pageView PDFGitHub2Add to collection

Community

Upload images, audio, and videos by dragging in the text input, pasting, orclicking here.

Tap or paste here to upload images

Get this paper in your agent:

hf papers read 2607\.19691

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper2

#### ModalityDance/slpo-coconut-gpt2 Text Generation• 0.1B• Updatedabout 23 hours ago • 49 #### ModalityDance/slpo-codi-gpt2 Text Generation• 0.1B• Updatedabout 23 hours ago • 51

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.19691 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.19691 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

Structured Role-Aware Policy Optimization for Multimodal Reasoning

arXiv cs.AI

This paper introduces Structured Role-Aware Policy Optimization (SRPO), a method that improves multimodal reasoning in Large Vision-Language Models by assigning token-level credit based on distinct perception and reasoning roles within reinforcement learning frameworks.