Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

Hugging Face Daily Papers Papers
llm post-training proxy-model reinforcement-learning modular-training signal-transfer weak-to-strong

Summary

Proposes PUST, a novel LLM post-training framework that decouples reward exploration from distribution alignment using a lightweight proxy model, enabling reusable update signals and efficient weak-to-strong enhancement across models.

Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.
Original Article
View Cached Full Text

Cached at: 07/14/26, 08:13 AM

Paper page - Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

Source: https://huggingface.co/papers/2607.11505 Published on Jul 13

·

Submitted byhttps://huggingface.co/fudaocheng

Fuon Jul 14

Abstract

Post-trainingisessentialforrefiningthedomain-specificcapabilitiesoflargelanguagemodels(LLMs),yetexistingrewardoptimizationanddistributionmatchingmethodstightlycouplepolicyexplorationwithdistributionalignment.Thiscouplingforcesexpensiveexplorationdirectlyonthepolicymodelandseverelyhinderstheasynchronousgeneration,reuse,andcross-modeltransferofoptimizationsignals.Inthispaper,weproposeProxy-guidedUpdateSignalTransfer(PUST),anovelpost-trainingframeworkthatfundamentallydecouplesupdate-signalexplorationfromdistributionalignment.Insteadofutilizingtheprimarymodelforcostlyexploration,PUSTemploysalightweightproxymodelasanefficienttestbedtodiscoverhigh-rewardbehaviors.Weextracttherelativeimprovementsignalbetweentheproxy’sinitialandoptimizedstates,transferringthisdirectionalupdatetotheprimarymodeltoguideitspolicyalignment.Thisdecoupledpipeline,comprisingproxyexploration,update-signalextraction,andsignaltransfer,significantlyreducescomputationaloverheadandenablesoptimizationsignalstobeasynchronouslygenerated,cached,andreused.Crucially,bytransferringrelativeimprovementsratherthanabsolutepolicydistributions,PUSTnaturallysupportsweak-to-strongimprovementandseamlesscross-modeltransfer.SystematicevaluationsonQwen3-familymodelsacrossmathandcodedomainsdemonstratethatupdatesignalsextractedfromsubstantiallyweakerproxiescanrobustlyandadjustablyenhancestrongerprimarymodels.Ultimately,PUSTtransformspost-trainingfromamonolithiconlineoptimizationprocessintoahighlymodular,reusable,andcost-efficientparadigm.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.11505 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.11505 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.11505 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Forecasting Downstream Performance of LLMs With Proxy Metrics

Hugging Face Daily Papers

This paper introduces proxy metrics based on token-level statistics from expert-written solutions to forecast downstream LLM performance, significantly outperforming loss-based methods in model selection, pretraining data selection, and training-time forecasting.

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Hugging Face Daily Papers

LLM-as-a-Tutor introduces a framework that extends LLM's role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.

Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

Hugging Face Daily Papers

This paper introduces 'progress advantage', an implicit advantage function derived from reinforcement learning post-training that enables effective step-level scoring for LLM agents without requiring dedicated reward model training. It outperforms confidence-based baselines and trained reward models across multiple benchmarks and model families.